2024-09-18T01:06:03.535 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T01:06:03.557 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T01:06:03.642 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706 branch: squid description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: ceph-qa@ceph.com first_in_suite: false job_id: '7909706' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 1983 sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f sleep_before_teardown: 0 subset: 79/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 targets: smithi066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGn6aYBEPHizdpZJ+69qOLn/mMp4MZTTPpuLz8kHDglD/5eyJWhiY+WovdoH4DdROl74VGgdBmBtT4aCUx/yNCU= smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbKvIrtgNKXW7OrMynfsNnBa2DwHxfRxcT/MU/kRBzd2q9W/ve+QPw3O5ahdvSfbu3T2PgayQrYyvuW1+ta3Fc= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T01:06:03.642 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa; will attempt to use it 2024-09-18T01:06:03.643 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa/tasks 2024-09-18T01:06:03.643 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T01:06:03.644 INFO:teuthology.task.internal:Checking packages... 2024-09-18T01:06:03.672 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c2ddf9721ef254645201cdbabd9c8f49c30a586f' 2024-09-18T01:06:03.672 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T01:06:03.672 INFO:teuthology.packaging:ref: None 2024-09-18T01:06:03.672 INFO:teuthology.packaging:tag: None 2024-09-18T01:06:03.672 INFO:teuthology.packaging:branch: squid 2024-09-18T01:06:03.672 INFO:teuthology.packaging:sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-18T01:06:03.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-18T01:06:03.891 INFO:teuthology.task.internal:Found packages for ceph version 19.2.0-0 2024-09-18T01:06:03.893 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T01:06:03.909 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T01:06:03.909 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T01:06:03.923 INFO:teuthology.task.internal:Saving configuration 2024-09-18T01:06:03.943 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T01:06:03.949 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T01:06:03.975 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 01:00:11.327932', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGn6aYBEPHizdpZJ+69qOLn/mMp4MZTTPpuLz8kHDglD/5eyJWhiY+WovdoH4DdROl74VGgdBmBtT4aCUx/yNCU='} 2024-09-18T01:06:04.000 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 01:00:11.329457', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbKvIrtgNKXW7OrMynfsNnBa2DwHxfRxcT/MU/kRBzd2q9W/ve+QPw3O5ahdvSfbu3T2PgayQrYyvuW1+ta3Fc='} 2024-09-18T01:06:04.000 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T01:06:04.010 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-18T01:06:04.010 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-18T01:06:04.010 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T01:06:04.091 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb72de7ac0>, signals=[15]) 2024-09-18T01:06:04.091 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T01:06:04.096 INFO:teuthology.task.internal:Opening connections... 2024-09-18T01:06:04.096 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:06:04.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:06:04.190 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:06:04.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:06:04.267 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T01:06:04.278 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2024-09-18T01:06:04.297 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2024-09-18T01:06:04.298 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream" 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9" 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9" 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9" 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2024-09-18T01:06:04.357 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon" 2024-09-18T01:06:04.358 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T01:06:04.358 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/" 2024-09-18T01:06:04.358 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T01:06:04.358 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T01:06:04.358 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T01:06:04.358 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2024-09-18T01:06:04.381 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-09-18T01:06:04.400 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-09-18T01:06:04.400 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-09-18T01:06:04.458 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="9" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="9" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el9" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:LOGO="fedora-logo-icon" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T01:06:04.459 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T01:06:04.460 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-09-18T01:06:04.481 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T01:06:04.493 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T01:06:04.502 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T01:06:04.503 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2024-09-18T01:06:04.505 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-09-18T01:06:04.522 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T01:06:04.527 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T01:06:04.528 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2024-09-18T01:06:04.564 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-09-18T01:06:04.596 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T01:06:04.653 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T01:06:04.670 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T01:06:04.670 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T01:06:04.671 DEBUG:teuthology.orchestra.run.smithi066:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T01:06:04.672 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T01:06:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:06:04.689 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-09-18T01:06:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:06:04.690 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-09-18T01:06:04.747 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-503.el9.x86_64 2024-09-18T01:06:04.747 INFO:teuthology.task.kernel:Running kernel on smithi115: 5.14.0-503.el9.x86_64 2024-09-18T01:06:04.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2024-09-18T01:06:04.749 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-503.el9.x86_64 2024-09-18T01:06:04.749 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.14.0-503.el9.x86_64 2024-09-18T01:06:04.749 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum install -y kernel 2024-09-18T01:06:07.114 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-18T01:06:08.213 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-18T01:06:08.807 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 35 MB/s | 20 MB 00:00 2024-09-18T01:06:13.172 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 20 MB 00:03 2024-09-18T01:06:13.508 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-18T01:06:15.992 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 19 kB 00:01 2024-09-18T01:06:17.687 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-18T01:06:18.339 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.5 MB 00:01 2024-09-18T01:06:19.840 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-18T01:06:21.585 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-18T01:06:23.277 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-09-18T01:06:25.266 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T01:06:25.267 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T01:06:25.354 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T01:06:25.363 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T01:06:25.364 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:06:25.364 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:06:25.364 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:06:25.364 INFO:teuthology.orchestra.run.smithi066.stdout:Install 4 Packages 2024-09-18T01:06:25.364 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:06:25.364 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 79 M 2024-09-18T01:06:25.365 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 126 M 2024-09-18T01:06:25.365 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:06:25.536 INFO:teuthology.orchestra.run.smithi066.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 299 kB/s | 36 kB 00:00 2024-09-18T01:06:26.940 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-18T01:06:28.880 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T01:06:28.880 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T01:06:28.971 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:06:28.975 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:06:28.975 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-09-18T01:06:28.975 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:06:28.975 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:06:28.975 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T01:06:28.975 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:06:28.976 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T01:06:28.976 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T01:06:28.976 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T01:06:28.976 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:06:28.976 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:06:28.976 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:06:28.976 INFO:teuthology.orchestra.run.smithi115.stdout:Install 4 Packages 2024-09-18T01:06:28.976 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:06:28.977 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 79 M 2024-09-18T01:06:28.977 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 126 M 2024-09-18T01:06:28.977 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:06:29.075 INFO:teuthology.orchestra.run.smithi115.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 460 kB/s | 36 kB 00:00 2024-09-18T01:06:29.676 INFO:teuthology.orchestra.run.smithi115.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-09-18T01:06:30.211 INFO:teuthology.orchestra.run.smithi066.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 7.2 MB/s | 35 MB 00:04 2024-09-18T01:06:30.267 INFO:teuthology.orchestra.run.smithi115.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-18T01:06:30.628 INFO:teuthology.orchestra.run.smithi066.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-09-18T01:06:30.776 INFO:teuthology.orchestra.run.smithi115.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 19 MB/s | 35 MB 00:01 2024-09-18T01:06:30.776 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:06:30.776 INFO:teuthology.orchestra.run.smithi115.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-18T01:06:31.212 INFO:teuthology.orchestra.run.smithi066.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 5.0 MB/s | 29 MB 00:05 2024-09-18T01:06:31.212 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:06:31.212 INFO:teuthology.orchestra.run.smithi066.stdout:Total 13 MB/s | 79 MB 00:05 2024-09-18T01:06:31.460 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:06:31.796 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:06:31.797 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:06:31.926 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:06:32.317 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:06:32.317 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:06:32.740 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:06:32.740 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:06:33.337 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:06:33.337 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:06:35.185 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:06:35.893 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T01:06:35.915 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:06:35.976 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T01:06:36.565 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T01:06:36.645 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T01:06:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T01:06:38.871 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T01:06:39.534 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T01:06:39.663 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T01:06:43.686 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T01:06:43.788 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:06:44.887 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T01:06:44.931 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:06:48.635 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:06:50.175 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:26.840 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:26.880 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:27.214 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:27.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T01:07:27.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T01:07:27.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:07:28.071 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:07:28.430 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T01:07:28.526 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T01:07:28.526 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T01:07:28.526 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T01:07:28.526 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T01:07:28.526 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T01:07:28.527 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-18T01:07:28.527 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T01:07:28.527 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T01:07:28.585 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T01:07:28.585 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T01:07:28.585 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2024-09-18T01:07:28.620 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.06 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 82.el9 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Group : Unspecified 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 8274670 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : builder@centos.org 2024-09-18T01:07:28.621 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:07:28.622 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T01:07:28.623 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T01:07:28.623 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 5.14.0-508.el9.x86_64 2024-09-18T01:07:28.624 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T01:07:29.267 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2024-09-18T01:07:30.113 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:30.155 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:30.510 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:30.510 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T01:07:30.510 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T01:07:30.510 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T01:07:30.614 INFO:teuthology.orchestra.run.smithi115.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T01:07:30.635 INFO:teuthology.orchestra.run.smithi115.stderr:done 2024-09-18T01:07:30.636 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-09-18T01:07:30.655 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.lpBq9zUMql 2024-09-18T01:07:30.655 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lpBq9zUMql 2024-09-18T01:07:30.730 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.lpBq9zUMql 2024-09-18T01:07:30.874 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.lpBq9zUMql is 6KB 2024-09-18T01:07:30.922 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.lpBq9zUMql 2024-09-18T01:07:30.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T01:07:30.999 INFO:teuthology.orchestra.run.smithi115.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T01:07:31.000 INFO:teuthology.orchestra.run.smithi115.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T01:07:31.000 INFO:teuthology.orchestra.run.smithi115.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T01:07:31.000 INFO:teuthology.orchestra.run.smithi115.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T01:07:31.001 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T01:07:31.142 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:07:31.407 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:07:31.749 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T01:07:31.829 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T01:07:31.829 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T01:07:31.829 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T01:07:31.830 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T01:07:31.830 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T01:07:31.830 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-18T01:07:31.830 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T01:07:31.830 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T01:07:31.892 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T01:07:31.892 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T01:07:31.892 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -qi grub2-tools 2024-09-18T01:07:31.934 INFO:teuthology.orchestra.run.smithi066.stdout:Name : grub2-tools 2024-09-18T01:07:31.934 INFO:teuthology.orchestra.run.smithi066.stdout:Epoch : 1 2024-09-18T01:07:31.934 INFO:teuthology.orchestra.run.smithi066.stdout:Version : 2.06 2024-09-18T01:07:31.934 INFO:teuthology.orchestra.run.smithi066.stdout:Release : 82.el9 2024-09-18T01:07:31.934 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: x86_64 2024-09-18T01:07:31.934 INFO:teuthology.orchestra.run.smithi066.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Group : Unspecified 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Size : 8274670 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:License : GPLv3+ 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Packager : builder@centos.org 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Vendor : CentOS 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T01:07:31.935 INFO:teuthology.orchestra.run.smithi066.stdout:Summary : Support tools for GRUB. 2024-09-18T01:07:31.936 INFO:teuthology.orchestra.run.smithi066.stdout:Description : 2024-09-18T01:07:31.936 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:07:31.936 INFO:teuthology.orchestra.run.smithi066.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T01:07:31.936 INFO:teuthology.orchestra.run.smithi066.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T01:07:31.936 INFO:teuthology.orchestra.run.smithi066.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T01:07:31.936 INFO:teuthology.orchestra.run.smithi066.stdout:hardware devices. 2024-09-18T01:07:31.936 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:07:31.936 INFO:teuthology.orchestra.run.smithi066.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T01:07:31.937 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T01:07:31.937 INFO:teuthology.task.kernel:Updating grub on smithi066 to boot 5.14.0-508.el9.x86_64 2024-09-18T01:07:31.937 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T01:07:32.576 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2024-09-18T01:07:33.928 INFO:teuthology.orchestra.run.smithi066.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T01:07:33.960 INFO:teuthology.orchestra.run.smithi066.stderr:done 2024-09-18T01:07:33.961 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-09-18T01:07:33.977 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.Wh2qew9zge 2024-09-18T01:07:33.977 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Wh2qew9zge 2024-09-18T01:07:34.044 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.Wh2qew9zge 2024-09-18T01:07:34.182 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.Wh2qew9zge is 6KB 2024-09-18T01:07:34.230 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.Wh2qew9zge 2024-09-18T01:07:34.246 DEBUG:teuthology.orchestra.run.smithi066:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T01:07:34.310 INFO:teuthology.orchestra.run.smithi066.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T01:07:34.311 INFO:teuthology.orchestra.run.smithi066.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T01:07:34.311 INFO:teuthology.orchestra.run.smithi066.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T01:07:34.311 INFO:teuthology.orchestra.run.smithi066.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T01:07:34.311 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T01:07:34.458 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2024-09-18T01:08:01.169 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-18T01:08:01.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T01:08:01.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:08:04.463 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-18T01:08:04.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-18T01:08:04.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:08:19.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-09-18T01:08:22.973 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-09-18T01:08:28.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T01:08:28.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:08:31.709 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-09-18T01:08:31.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-18T01:08:31.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:08:35.041 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-09-18T01:08:43.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T01:08:43.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:08:47.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-18T01:08:47.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:08:50.109 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-09-18T01:09:02.141 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-09-18T01:09:05.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-18T01:09:05.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:09:17.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T01:09:17.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:09:17.662 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-09-18T01:09:18.041 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T01:09:18.041 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-18T01:09:18.041 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-09-18T01:09:18.056 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-508.el9.x86_64 2024-09-18T01:09:18.056 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T01:09:18.056 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T01:09:18.056 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T01:09:19.057 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-18T01:09:19.058 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T01:09:19.195 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2024-09-18T01:09:19.221 DEBUG:teuthology.parallel:result is None 2024-09-18T01:09:37.192 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-09-18T01:09:37.701 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-18T01:09:37.702 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-18T01:09:37.702 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-09-18T01:09:37.717 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-508.el9.x86_64 2024-09-18T01:09:37.717 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T01:09:37.717 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T01:09:37.717 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T01:09:38.719 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-18T01:09:38.719 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T01:09:38.850 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2024-09-18T01:09:38.881 DEBUG:teuthology.parallel:result is None 2024-09-18T01:09:38.881 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T01:09:38.888 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T01:09:38.888 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T01:09:38.890 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T01:09:38.909 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T01:09:38.919 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T01:09:38.925 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T01:09:38.925 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T01:09:38.945 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T01:09:39.009 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T01:09:39.016 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T01:09:39.016 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T01:09:39.018 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T01:09:39.052 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T01:09:39.053 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T01:09:39.063 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T01:09:39.065 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T01:09:39.066 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T01:09:39.119 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T01:09:39.119 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T01:09:39.121 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T01:09:39.148 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T01:09:39.170 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T01:09:39.170 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T01:09:39.189 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T01:09:39.205 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T01:09:39.288 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T01:09:39.349 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T01:09:39.389 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T01:09:39.435 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:09:39.435 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T01:09:39.502 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T01:09:39.543 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T01:09:39.592 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T01:09:39.669 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T01:09:39.718 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:09:39.719 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T01:09:39.785 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2024-09-18T01:09:39.787 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-09-18T01:09:39.832 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T01:09:39.873 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T01:09:40.298 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T01:09:40.309 INFO:teuthology.task.internal:Starting timer... 2024-09-18T01:09:40.309 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T01:09:40.320 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T01:09:40.327 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-18T01:09:40.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo service auditd rotate 2024-09-18T01:09:40.375 INFO:teuthology.orchestra.run.smithi066.stdout:Rotating logs: 2024-09-18T01:09:40.376 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2024-09-18T01:09:40.432 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2024-09-18T01:09:40.434 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T01:09:40.435 DEBUG:teuthology.orchestra.run.smithi066:> /usr/sbin/getenforce 2024-09-18T01:09:40.463 INFO:teuthology.orchestra.run.smithi066.stdout:Permissive 2024-09-18T01:09:40.463 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2024-09-18T01:09:40.489 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2024-09-18T01:09:40.489 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi066.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi115.front.sepia.ceph.com': 'permissive'} 2024-09-18T01:09:40.490 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T01:09:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:09:40.531 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T01:09:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:09:40.557 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T01:09:40.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo /usr/sbin/setenforce permissive 2024-09-18T01:09:40.609 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2024-09-18T01:09:40.640 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T01:09:40.649 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T01:09:40.660 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T01:09:40.661 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi066.front.sepia.ceph.com,smithi115.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T01:13:25.123 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi066.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com')] 2024-09-18T01:13:25.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-18T01:13:25.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:13:25.208 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-09-18T01:13:25.285 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-09-18T01:13:25.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T01:13:25.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T01:13:25.360 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-09-18T01:13:25.445 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T01:13:25.445 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T01:13:25.458 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T01:13:25.458 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T01:13:25.459 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T01:13:25.461 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T01:13:25.461 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T01:13:25.490 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T01:13:25.507 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T01:13:25.515 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T01:13:25.530 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T01:13:25.533 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: ntpd: command not found 2024-09-18T01:13:25.545 INFO:teuthology.orchestra.run.smithi066.stdout:506 Cannot talk to daemon 2024-09-18T01:13:25.554 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2024-09-18T01:13:25.559 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T01:13:25.565 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2024-09-18T01:13:25.574 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T01:13:25.580 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T01:13:25.594 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T01:13:25.633 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-09-18T01:13:25.636 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T01:13:25.636 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-09-18T01:13:25.637 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:13:25.637 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:13:25.637 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:13:25.637 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:13:25.661 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-09-18T01:13:25.664 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T01:13:25.664 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-09-18T01:13:25.664 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:13:25.665 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:13:25.665 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:13:25.665 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:13:25.666 INFO:teuthology.run_tasks:Running task install... 2024-09-18T01:13:25.673 DEBUG:teuthology.task.install:project ceph 2024-09-18T01:13:25.674 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}} 2024-09-18T01:13:25.674 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'} 2024-09-18T01:13:25.674 INFO:teuthology.task.install:Using flavor: default 2024-09-18T01:13:25.679 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T01:13:25.680 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-18T01:13:25.680 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'tag': None, 'wait_for_package': False} 2024-09-18T01:13:25.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-18T01:13:25.682 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'tag': None, 'wait_for_package': False} 2024-09-18T01:13:25.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-18T01:13:25.826 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/ 2024-09-18T01:13:25.826 INFO:teuthology.task.install.rpm:Package version is 19.1.1-330.gc2ddf972 2024-09-18T01:13:25.830 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/ 2024-09-18T01:13:25.830 INFO:teuthology.task.install.rpm:Package version is 19.1.1-330.gc2ddf972 2024-09-18T01:13:25.938 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T01:13:25.938 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:13:25.938 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T01:13:25.946 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T01:13:25.946 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:13:25.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T01:13:25.964 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-18T01:13:25.964 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c2ddf9721ef254645201cdbabd9c8f49c30a586f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T01:13:25.974 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-18T01:13:25.974 DEBUG:teuthology.orchestra.run.smithi066:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c2ddf9721ef254645201cdbabd9c8f49c30a586f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T01:13:26.030 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T01:13:26.040 DEBUG:teuthology.orchestra.run.smithi066:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T01:13:26.106 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T01:13:26.116 DEBUG:teuthology.orchestra.run.smithi066:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T01:13:26.169 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2024-09-18T01:13:26.171 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2024-09-18T01:13:26.181 INFO:teuthology.orchestra.run.smithi066.stdout:check_obsoletes = 1 2024-09-18T01:13:26.183 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum clean all 2024-09-18T01:13:26.409 INFO:teuthology.orchestra.run.smithi115.stdout:45 files removed 2024-09-18T01:13:26.418 INFO:teuthology.orchestra.run.smithi066.stdout:45 files removed 2024-09-18T01:13:26.435 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw 2024-09-18T01:13:26.445 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-radosgw 2024-09-18T01:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 299 kB/s | 84 kB 00:00 2024-09-18T01:13:27.010 INFO:teuthology.orchestra.run.smithi066.stdout:ceph packages for x86_64 313 kB/s | 84 kB 00:00 2024-09-18T01:13:27.281 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-18T01:13:27.303 INFO:teuthology.orchestra.run.smithi066.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-18T01:13:27.565 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 7.6 kB/s | 1.9 kB 00:00 2024-09-18T01:13:27.578 INFO:teuthology.orchestra.run.smithi066.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-18T01:13:28.945 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.3 MB 00:01 2024-09-18T01:13:31.206 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.3 MB 00:03 2024-09-18T01:13:33.513 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 20 MB 00:03 2024-09-18T01:13:33.651 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-18T01:13:37.888 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-18T01:13:39.285 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-18T01:13:39.414 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 19 kB 00:00 2024-09-18T01:13:40.894 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 19 kB 00:00 2024-09-18T01:13:41.285 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 13 MB/s | 23 MB 00:01 2024-09-18T01:13:42.506 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-18T01:13:46.832 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-18T01:13:47.861 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-18T01:13:49.112 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:13:49.124 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:13:49.124 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:13:49.124 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 11 M 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrading: 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout: librados2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.4 M 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.2 M 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.4 M 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 21 M 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 25 k 2024-09-18T01:13:49.125 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 751 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 506 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.3 M 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 45 k 2024-09-18T01:13:49.126 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 136 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 162 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 320 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 300 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 99 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout:Installing weak dependencies: 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:13:49.127 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:13:49.128 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:13:49.128 INFO:teuthology.orchestra.run.smithi066.stdout:Install 30 Packages 2024-09-18T01:13:49.128 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrade 2 Packages 2024-09-18T01:13:49.128 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:13:49.128 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 60 M 2024-09-18T01:13:49.128 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:13:49.917 INFO:teuthology.orchestra.run.smithi066.stdout:(1/32): ceph-base-19.1.1-330.gc2ddf972.el9.x86_ 9.0 MB/s | 5.4 MB 00:00 2024-09-18T01:13:49.984 INFO:teuthology.orchestra.run.smithi066.stdout:(2/32): ceph-selinux-19.1.1-330.gc2ddf972.el9.x 376 kB/s | 25 kB 00:00 2024-09-18T01:13:50.101 INFO:teuthology.orchestra.run.smithi066.stdout:(3/32): ceph-radosgw-19.1.1-330.gc2ddf972.el9.x 14 MB/s | 11 MB 00:00 2024-09-18T01:13:50.113 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 11 M 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.4 M 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.2 M 2024-09-18T01:13:50.127 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.4 M 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 21 M 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 25 k 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 751 k 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T01:13:50.128 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 506 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.3 M 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 45 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 136 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 162 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 320 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 300 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 99 k 2024-09-18T01:13:50.129 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout:Installing weak dependencies: 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout:Install 30 Packages 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 2 Packages 2024-09-18T01:13:50.130 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:13:50.131 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 60 M 2024-09-18T01:13:50.131 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:13:50.134 INFO:teuthology.orchestra.run.smithi066.stdout:(4/32): libcephfs2-19.1.1-330.gc2ddf972.el9.x86 4.9 MB/s | 751 kB 00:00 2024-09-18T01:13:50.176 INFO:teuthology.orchestra.run.smithi066.stdout:(5/32): libradosstriper1-19.1.1-330.gc2ddf972.e 6.6 MB/s | 506 kB 00:00 2024-09-18T01:13:50.243 INFO:teuthology.orchestra.run.smithi066.stdout:(6/32): python3-ceph-argparse-19.1.1-330.gc2ddf 678 kB/s | 45 kB 00:00 2024-09-18T01:13:50.335 INFO:teuthology.orchestra.run.smithi066.stdout:(7/32): python3-ceph-common-19.1.1-330.gc2ddf97 1.5 MB/s | 136 kB 00:00 2024-09-18T01:13:50.419 INFO:teuthology.orchestra.run.smithi066.stdout:(8/32): librgw2-19.1.1-330.gc2ddf972.el9.x86_64 19 MB/s | 5.3 MB 00:00 2024-09-18T01:13:50.444 INFO:teuthology.orchestra.run.smithi066.stdout:(9/32): python3-cephfs-19.1.1-330.gc2ddf972.el9 1.5 MB/s | 162 kB 00:00 2024-09-18T01:13:50.469 INFO:teuthology.orchestra.run.smithi066.stdout:(10/32): python3-rados-19.1.1-330.gc2ddf972.el9 6.2 MB/s | 320 kB 00:00 2024-09-18T01:13:50.494 INFO:teuthology.orchestra.run.smithi066.stdout:(11/32): python3-rbd-19.1.1-330.gc2ddf972.el9.x 5.8 MB/s | 300 kB 00:00 2024-09-18T01:13:50.686 INFO:teuthology.orchestra.run.smithi066.stdout:(12/32): ceph-common-19.1.1-330.gc2ddf972.el9.x 15 MB/s | 21 MB 00:01 2024-09-18T01:13:50.711 INFO:teuthology.orchestra.run.smithi066.stdout:(13/32): python3-rgw-19.1.1-330.gc2ddf972.el9.x 410 kB/s | 99 kB 00:00 2024-09-18T01:13:50.812 INFO:teuthology.orchestra.run.smithi066.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 453 kB/s | 45 kB 00:00 2024-09-18T01:13:50.837 INFO:teuthology.orchestra.run.smithi066.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 312 kB/s | 107 kB 00:00 2024-09-18T01:13:50.863 INFO:teuthology.orchestra.run.smithi066.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 912 kB/s | 160 kB 00:00 2024-09-18T01:13:50.913 INFO:teuthology.orchestra.run.smithi066.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 848 kB/s | 42 kB 00:00 2024-09-18T01:13:50.938 INFO:teuthology.orchestra.run.smithi066.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-18T01:13:50.946 INFO:teuthology.orchestra.run.smithi115.stdout:(1/32): ceph-base-19.1.1-330.gc2ddf972.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-18T01:13:50.963 INFO:teuthology.orchestra.run.smithi066.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-18T01:13:50.989 INFO:teuthology.orchestra.run.smithi066.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 714 kB/s | 36 kB 00:00 2024-09-18T01:13:50.996 INFO:teuthology.orchestra.run.smithi115.stdout:(2/32): ceph-selinux-19.1.1-330.gc2ddf972.el9.x 503 kB/s | 25 kB 00:00 2024-09-18T01:13:51.014 INFO:teuthology.orchestra.run.smithi066.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-09-18T01:13:51.039 INFO:teuthology.orchestra.run.smithi066.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-18T01:13:51.088 INFO:teuthology.orchestra.run.smithi115.stdout:(3/32): libcephfs2-19.1.1-330.gc2ddf972.el9.x86 8.0 MB/s | 751 kB 00:00 2024-09-18T01:13:51.106 INFO:teuthology.orchestra.run.smithi066.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 513 kB/s | 26 kB 00:00 2024-09-18T01:13:51.131 INFO:teuthology.orchestra.run.smithi066.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-18T01:13:51.156 INFO:teuthology.orchestra.run.smithi066.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2024-09-18T01:13:51.171 INFO:teuthology.orchestra.run.smithi115.stdout:(4/32): libradosstriper1-19.1.1-330.gc2ddf972.e 6.0 MB/s | 506 kB 00:00 2024-09-18T01:13:51.182 INFO:teuthology.orchestra.run.smithi066.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-18T01:13:51.232 INFO:teuthology.orchestra.run.smithi066.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-18T01:13:51.340 INFO:teuthology.orchestra.run.smithi066.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-09-18T01:13:51.480 INFO:teuthology.orchestra.run.smithi115.stdout:(5/32): ceph-radosgw-19.1.1-330.gc2ddf972.el9.x 10 MB/s | 11 MB 00:01 2024-09-18T01:13:51.516 INFO:teuthology.orchestra.run.smithi066.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-09-18T01:13:51.599 INFO:teuthology.orchestra.run.smithi066.stdout:(30/32): librados2-19.1.1-330.gc2ddf972.el9.x86 13 MB/s | 3.4 MB 00:00 2024-09-18T01:13:51.689 INFO:teuthology.orchestra.run.smithi115.stdout:(6/32): ceph-common-19.1.1-330.gc2ddf972.el9.x8 17 MB/s | 21 MB 00:01 2024-09-18T01:13:51.714 INFO:teuthology.orchestra.run.smithi115.stdout:(7/32): python3-ceph-argparse-19.1.1-330.gc2ddf 193 kB/s | 45 kB 00:00 2024-09-18T01:13:51.724 INFO:teuthology.orchestra.run.smithi066.stdout:(31/32): libarrow-9.0.0-11.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-09-18T01:13:51.739 INFO:teuthology.orchestra.run.smithi115.stdout:(8/32): python3-ceph-common-19.1.1-330.gc2ddf97 2.7 MB/s | 136 kB 00:00 2024-09-18T01:13:51.773 INFO:teuthology.orchestra.run.smithi115.stdout:(9/32): python3-cephfs-19.1.1-330.gc2ddf972.el9 2.7 MB/s | 162 kB 00:00 2024-09-18T01:13:51.775 INFO:teuthology.orchestra.run.smithi066.stdout:(32/32): librbd1-19.1.1-330.gc2ddf972.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-18T01:13:51.780 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:13:51.780 INFO:teuthology.orchestra.run.smithi066.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-18T01:13:51.849 INFO:teuthology.orchestra.run.smithi115.stdout:(10/32): librgw2-19.1.1-330.gc2ddf972.el9.x86_6 7.9 MB/s | 5.3 MB 00:00 2024-09-18T01:13:51.873 INFO:teuthology.orchestra.run.smithi115.stdout:(11/32): python3-rados-19.1.1-330.gc2ddf972.el9 2.3 MB/s | 320 kB 00:00 2024-09-18T01:13:51.887 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:13:51.907 INFO:teuthology.orchestra.run.smithi115.stdout:(12/32): python3-rbd-19.1.1-330.gc2ddf972.el9.x 2.2 MB/s | 300 kB 00:00 2024-09-18T01:13:51.927 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:13:51.927 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:13:51.932 INFO:teuthology.orchestra.run.smithi115.stdout:(13/32): python3-rgw-19.1.1-330.gc2ddf972.el9.x 1.2 MB/s | 99 kB 00:00 2024-09-18T01:13:52.025 INFO:teuthology.orchestra.run.smithi115.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 709 kB/s | 107 kB 00:00 2024-09-18T01:13:52.050 INFO:teuthology.orchestra.run.smithi115.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 386 kB/s | 45 kB 00:00 2024-09-18T01:13:52.075 INFO:teuthology.orchestra.run.smithi115.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 957 kB/s | 160 kB 00:00 2024-09-18T01:13:52.117 INFO:teuthology.orchestra.run.smithi115.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-18T01:13:52.142 INFO:teuthology.orchestra.run.smithi115.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-18T01:13:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-18T01:13:52.193 INFO:teuthology.orchestra.run.smithi115.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.8 MB/s | 221 kB 00:00 2024-09-18T01:13:52.218 INFO:teuthology.orchestra.run.smithi115.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-18T01:13:52.243 INFO:teuthology.orchestra.run.smithi115.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-18T01:13:52.302 INFO:teuthology.orchestra.run.smithi115.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-09-18T01:13:52.327 INFO:teuthology.orchestra.run.smithi115.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-18T01:13:52.353 INFO:teuthology.orchestra.run.smithi115.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-09-18T01:13:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-18T01:13:52.437 INFO:teuthology.orchestra.run.smithi115.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-18T01:13:52.487 INFO:teuthology.orchestra.run.smithi115.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-09-18T01:13:52.613 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:13:52.613 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:13:52.645 INFO:teuthology.orchestra.run.smithi115.stdout:(29/32): librados2-19.1.1-330.gc2ddf972.el9.x86 22 MB/s | 3.4 MB 00:00 2024-09-18T01:13:52.696 INFO:teuthology.orchestra.run.smithi115.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-09-18T01:13:52.796 INFO:teuthology.orchestra.run.smithi115.stdout:(31/32): librbd1-19.1.1-330.gc2ddf972.el9.x86_6 21 MB/s | 3.2 MB 00:00 2024-09-18T01:13:52.888 INFO:teuthology.orchestra.run.smithi115.stdout:(32/32): libarrow-9.0.0-11.el9.x86_64.rpm 6.6 MB/s | 4.4 MB 00:00 2024-09-18T01:13:52.892 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:13:52.892 INFO:teuthology.orchestra.run.smithi115.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-18T01:13:52.995 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:13:53.030 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:13:53.031 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:13:53.710 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:13:53.710 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:13:54.190 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:13:54.268 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-18T01:13:54.297 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-18T01:13:54.322 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-18T01:13:54.344 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-18T01:13:54.369 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-18T01:13:54.531 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-18T01:13:54.584 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-18T01:13:54.933 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-18T01:13:55.031 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-18T01:13:55.125 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-18T01:13:55.192 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:13:55.215 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 8/34 2024-09-18T01:13:55.250 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 9/34 2024-09-18T01:13:55.280 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-18T01:13:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 9/34 2024-09-18T01:13:55.310 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-18T01:13:55.335 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-18T01:13:55.342 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-18T01:13:55.366 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-18T01:13:55.388 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-18T01:13:55.394 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x 11/34 2024-09-18T01:13:55.465 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 12/34 2024-09-18T01:13:55.494 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-18T01:13:55.558 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-18T01:13:55.585 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-18T01:13:55.592 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-18T01:13:55.646 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-18T01:13:55.703 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-18T01:13:55.735 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-18T01:13:55.775 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-18T01:13:55.816 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-18T01:13:55.846 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-18T01:13:55.885 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-18T01:13:56.036 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-18T01:13:56.103 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-18T01:13:56.193 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-18T01:13:56.226 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-18T01:13:56.266 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 22/34 2024-09-18T01:13:56.282 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 8/34 2024-09-18T01:13:56.313 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 9/34 2024-09-18T01:13:56.344 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 22/34 2024-09-18T01:13:56.371 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 23/34 2024-09-18T01:13:56.375 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 9/34 2024-09-18T01:13:56.404 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-18T01:13:56.415 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-18T01:13:56.448 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x 11/34 2024-09-18T01:13:56.516 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 12/34 2024-09-18T01:13:56.548 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-18T01:13:56.621 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-18T01:13:56.663 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-18T01:13:56.772 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-18T01:13:56.806 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-18T01:13:56.853 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-18T01:13:56.888 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-18T01:13:56.917 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-18T01:13:56.925 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-18T01:13:56.950 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-18T01:13:56.952 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 26/34 2024-09-18T01:13:57.024 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 26/34 2024-09-18T01:13:57.096 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 27/34 2024-09-18T01:13:57.129 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86 28/34 2024-09-18T01:13:57.276 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-18T01:13:57.313 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 22/34 2024-09-18T01:13:57.413 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 22/34 2024-09-18T01:13:57.443 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 23/34 2024-09-18T01:13:57.487 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-18T01:13:57.996 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-18T01:13:58.024 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 26/34 2024-09-18T01:13:58.095 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 26/34 2024-09-18T01:13:58.172 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 27/34 2024-09-18T01:13:58.201 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86 28/34 2024-09-18T01:13:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-18T01:13:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-18T01:14:00.182 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-18T01:14:00.220 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 30/34 2024-09-18T01:14:00.305 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 30/34 2024-09-18T01:14:00.305 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T01:14:00.305 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T01:14:00.306 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:00.346 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-18T01:14:00.456 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-18T01:14:00.489 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-18T01:14:01.049 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-18T01:14:01.084 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 30/34 2024-09-18T01:14:01.175 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 30/34 2024-09-18T01:14:01.175 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T01:14:01.176 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T01:14:01.176 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:01.210 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-18T01:14:10.817 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-18T01:14:10.817 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /sys 2024-09-18T01:14:10.817 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /proc 2024-09-18T01:14:10.817 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /mnt 2024-09-18T01:14:10.817 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /var/tmp 2024-09-18T01:14:10.817 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /home 2024-09-18T01:14:10.817 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /root 2024-09-18T01:14:10.818 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /tmp 2024-09-18T01:14:10.818 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:11.247 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-18T01:14:11.247 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2024-09-18T01:14:11.247 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2024-09-18T01:14:11.247 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2024-09-18T01:14:11.247 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2024-09-18T01:14:11.247 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2024-09-18T01:14:11.247 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /root 2024-09-18T01:14:11.247 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2024-09-18T01:14:11.248 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:11.839 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 32/34 2024-09-18T01:14:11.867 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 32/34 2024-09-18T01:14:11.867 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:11.868 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T01:14:11.868 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T01:14:11.868 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T01:14:11.868 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:11.869 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-18T01:14:11.958 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-18T01:14:11.958 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-18T01:14:12.202 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 32/34 2024-09-18T01:14:12.229 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 32/34 2024-09-18T01:14:12.229 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:12.229 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T01:14:12.229 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T01:14:12.229 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T01:14:12.229 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:12.231 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-18T01:14:12.330 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-18T01:14:12.330 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-18T01:14:13.176 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-18T01:14:13.176 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 1/34 2024-09-18T01:14:13.176 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2/34 2024-09-18T01:14:13.176 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 3/34 2024-09-18T01:14:13.176 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 4/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 5/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 6/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x 8/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86 9/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 10/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 11/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 12/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-18T01:14:13.177 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-18T01:14:13.178 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-18T01:14:13.179 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-18T01:14:13.179 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-18T01:14:13.179 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-18T01:14:13.179 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-18T01:14:13.179 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-18T01:14:13.180 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 33/34 2024-09-18T01:14:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-18T01:14:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 1/34 2024-09-18T01:14:13.585 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 3/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 4/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 5/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 6/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x 8/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86 9/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 10/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 11/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 12/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-18T01:14:13.586 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-18T01:14:13.587 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-18T01:14:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-18T01:14:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-18T01:14:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-18T01:14:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-18T01:14:13.588 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-18T01:14:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-18T01:14:13.589 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 33/34 2024-09-18T01:14:14.100 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-18T01:14:14.100 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:14.100 INFO:teuthology.orchestra.run.smithi066.stdout:Upgraded: 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.101 INFO:teuthology.orchestra.run.smithi066.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T01:14:14.102 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout: re2-1:20211101-20.el9.x86_64 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:14.103 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:14:14.503 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-test 2024-09-18T01:14:14.723 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-18T01:14:14.723 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:14.723 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.724 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T01:14:14.725 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout: re2-1:20211101-20.el9.x86_64 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:14.726 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:14:15.072 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-test 2024-09-18T01:14:15.110 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:29 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:14:15.253 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:14:15.254 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:15.254 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:14:15.254 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:15.254 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:14:15.254 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 49 M 2024-09-18T01:14:15.254 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:14:15.254 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 163 k 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout:Install 4 Packages 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 49 M 2024-09-18T01:14:15.255 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 209 M 2024-09-18T01:14:15.256 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:14:15.587 INFO:teuthology.orchestra.run.smithi066.stdout:(1/4): libcephsqlite-19.1.1-330.gc2ddf972.el9.x 814 kB/s | 163 kB 00:00 2024-09-18T01:14:15.654 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:28 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:14:15.672 INFO:teuthology.orchestra.run.smithi066.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.0 MB/s | 304 kB 00:00 2024-09-18T01:14:15.755 INFO:teuthology.orchestra.run.smithi066.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 381 kB/s | 64 kB 00:00 2024-09-18T01:14:15.795 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:14:15.796 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:15.796 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:14:15.796 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:15.796 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 49 M 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 163 k 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout:Install 4 Packages 2024-09-18T01:14:15.797 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:15.798 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 49 M 2024-09-18T01:14:15.798 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 209 M 2024-09-18T01:14:15.798 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:14:16.095 INFO:teuthology.orchestra.run.smithi115.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-18T01:14:16.146 INFO:teuthology.orchestra.run.smithi115.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-18T01:14:16.213 INFO:teuthology.orchestra.run.smithi115.stdout:(3/4): libcephsqlite-19.1.1-330.gc2ddf972.el9.x 447 kB/s | 163 kB 00:00 2024-09-18T01:14:16.955 INFO:teuthology.orchestra.run.smithi066.stdout:(4/4): ceph-test-19.1.1-330.gc2ddf972.el9.x86_6 31 MB/s | 49 MB 00:01 2024-09-18T01:14:16.957 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:16.957 INFO:teuthology.orchestra.run.smithi066.stdout:Total 29 MB/s | 49 MB 00:01 2024-09-18T01:14:16.986 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:14:17.008 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:14:17.008 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:14:17.422 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:14:17.422 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:14:17.975 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:14:18.058 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T01:14:18.104 INFO:teuthology.orchestra.run.smithi115.stdout:(4/4): ceph-test-19.1.1-330.gc2ddf972.el9.x86_6 22 MB/s | 49 MB 00:02 2024-09-18T01:14:18.105 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T01:14:18.106 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:18.107 INFO:teuthology.orchestra.run.smithi115.stdout:Total 21 MB/s | 49 MB 00:02 2024-09-18T01:14:18.131 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 3/4 2024-09-18T01:14:18.134 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:14:18.154 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:14:18.154 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:14:18.546 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:14:18.546 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:14:19.083 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:14:19.174 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T01:14:19.220 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T01:14:19.248 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 3/4 2024-09-18T01:14:23.058 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 3/4 2024-09-18T01:14:23.141 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 4/4 2024-09-18T01:14:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 3/4 2024-09-18T01:14:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 4/4 2024-09-18T01:14:25.453 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 4/4 2024-09-18T01:14:25.453 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 1/4 2024-09-18T01:14:25.453 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2/4 2024-09-18T01:14:25.454 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T01:14:25.755 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T01:14:25.755 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:25.755 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:14:25.756 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:25.756 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:25.756 INFO:teuthology.orchestra.run.smithi066.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T01:14:25.756 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T01:14:25.756 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:25.756 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:14:26.024 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph 2024-09-18T01:14:26.522 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 4/4 2024-09-18T01:14:26.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 1/4 2024-09-18T01:14:26.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2/4 2024-09-18T01:14:26.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T01:14:26.643 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:40 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:14:26.789 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:14:26.791 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout: ceph x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 6.5 k 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 2.4 M 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 1.5 M 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 249 k 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 4.7 M 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 17 M 2024-09-18T01:14:26.792 INFO:teuthology.orchestra.run.smithi066.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T01:14:26.793 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T01:14:26.794 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout:Install 37 Packages 2024-09-18T01:14:26.795 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:26.796 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 30 M 2024-09-18T01:14:26.796 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 108 M 2024-09-18T01:14:26.796 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:14:26.847 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T01:14:26.848 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:26.848 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:14:26.848 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:26.848 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:26.848 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T01:14:26.848 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T01:14:26.848 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:26.848 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:14:27.141 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph 2024-09-18T01:14:27.727 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:40 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:14:27.772 INFO:teuthology.orchestra.run.smithi066.stdout:(1/37): ceph-19.1.1-330.gc2ddf972.el9.x86_64.rp 44 kB/s | 6.5 kB 00:00 2024-09-18T01:14:27.871 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:14:27.873 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:27.873 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 6.5 k 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 2.4 M 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 1.5 M 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 249 k 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 4.7 M 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 17 M 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-18T01:14:27.874 INFO:teuthology.orchestra.run.smithi115.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T01:14:27.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T01:14:27.876 INFO:teuthology.orchestra.run.smithi115.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T01:14:27.877 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T01:14:27.877 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T01:14:27.877 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T01:14:27.877 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:27.877 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:14:27.877 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:27.877 INFO:teuthology.orchestra.run.smithi115.stdout:Install 37 Packages 2024-09-18T01:14:27.877 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:27.878 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 30 M 2024-09-18T01:14:27.878 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 108 M 2024-09-18T01:14:27.878 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:14:27.972 INFO:teuthology.orchestra.run.smithi066.stdout:(2/37): ceph-mds-19.1.1-330.gc2ddf972.el9.x86_6 6.9 MB/s | 2.4 MB 00:00 2024-09-18T01:14:28.006 INFO:teuthology.orchestra.run.smithi066.stdout:(3/37): ceph-mgr-19.1.1-330.gc2ddf972.el9.x86_6 4.0 MB/s | 1.5 MB 00:00 2024-09-18T01:14:28.089 INFO:teuthology.orchestra.run.smithi066.stdout:(4/37): ceph-mgr-modules-core-19.1.1-330.gc2ddf 2.9 MB/s | 249 kB 00:00 2024-09-18T01:14:28.350 INFO:teuthology.orchestra.run.smithi115.stdout:(1/37): ceph-19.1.1-330.gc2ddf972.el9.x86_64.rp 27 kB/s | 6.5 kB 00:00 2024-09-18T01:14:28.481 INFO:teuthology.orchestra.run.smithi066.stdout:(5/37): ceph-osd-19.1.1-330.gc2ddf972.el9.x86_6 33 MB/s | 17 MB 00:00 2024-09-18T01:14:28.551 INFO:teuthology.orchestra.run.smithi115.stdout:(2/37): ceph-mds-19.1.1-330.gc2ddf972.el9.x86_6 5.4 MB/s | 2.4 MB 00:00 2024-09-18T01:14:28.573 INFO:teuthology.orchestra.run.smithi066.stdout:(6/37): ceph-mon-19.1.1-330.gc2ddf972.el9.x86_6 5.9 MB/s | 4.7 MB 00:00 2024-09-18T01:14:28.593 INFO:teuthology.orchestra.run.smithi115.stdout:(3/37): ceph-mgr-19.1.1-330.gc2ddf972.el9.x86_6 3.1 MB/s | 1.5 MB 00:00 2024-09-18T01:14:28.693 INFO:teuthology.orchestra.run.smithi115.stdout:(4/37): ceph-mgr-modules-core-19.1.1-330.gc2ddf 2.4 MB/s | 249 kB 00:00 2024-09-18T01:14:28.869 INFO:teuthology.orchestra.run.smithi115.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-18T01:14:28.935 INFO:teuthology.orchestra.run.smithi115.stdout:(6/37): ceph-mon-19.1.1-330.gc2ddf972.el9.x86_6 8.1 MB/s | 4.7 MB 00:00 2024-09-18T01:14:28.973 INFO:teuthology.orchestra.run.smithi066.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 286 kB/s | 253 kB 00:00 2024-09-18T01:14:29.127 INFO:teuthology.orchestra.run.smithi115.stdout:(7/37): ceph-osd-19.1.1-330.gc2ddf972.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-18T01:14:29.232 INFO:teuthology.orchestra.run.smithi066.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 162 kB/s | 106 kB 00:00 2024-09-18T01:14:29.253 INFO:teuthology.orchestra.run.smithi115.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 336 kB/s | 106 kB 00:00 2024-09-18T01:14:29.278 INFO:teuthology.orchestra.run.smithi115.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 901 kB/s | 135 kB 00:00 2024-09-18T01:14:29.312 INFO:teuthology.orchestra.run.smithi115.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-18T01:14:29.362 INFO:teuthology.orchestra.run.smithi115.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-09-18T01:14:29.387 INFO:teuthology.orchestra.run.smithi115.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-09-18T01:14:29.496 INFO:teuthology.orchestra.run.smithi115.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-18T01:14:29.521 INFO:teuthology.orchestra.run.smithi115.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 902 kB/s | 188 kB 00:00 2024-09-18T01:14:29.524 INFO:teuthology.orchestra.run.smithi066.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.2 MB/s | 1.2 MB 00:01 2024-09-18T01:14:29.546 INFO:teuthology.orchestra.run.smithi115.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 935 kB/s | 172 kB 00:00 2024-09-18T01:14:29.571 INFO:teuthology.orchestra.run.smithi115.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-18T01:14:29.597 INFO:teuthology.orchestra.run.smithi115.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 766 kB/s | 77 kB 00:00 2024-09-18T01:14:29.622 INFO:teuthology.orchestra.run.smithi115.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-18T01:14:29.664 INFO:teuthology.orchestra.run.smithi115.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-18T01:14:29.666 INFO:teuthology.orchestra.run.smithi066.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 195 kB/s | 135 kB 00:00 2024-09-18T01:14:29.689 INFO:teuthology.orchestra.run.smithi115.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-09-18T01:14:29.714 INFO:teuthology.orchestra.run.smithi115.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-09-18T01:14:29.748 INFO:teuthology.orchestra.run.smithi115.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-18T01:14:29.773 INFO:teuthology.orchestra.run.smithi115.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 206 kB/s | 11 kB 00:00 2024-09-18T01:14:29.790 INFO:teuthology.orchestra.run.smithi115.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 438 kB/s | 18 kB 00:00 2024-09-18T01:14:29.791 INFO:teuthology.orchestra.run.smithi066.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-18T01:14:29.807 INFO:teuthology.orchestra.run.smithi115.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 690 kB/s | 23 kB 00:00 2024-09-18T01:14:29.824 INFO:teuthology.orchestra.run.smithi115.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-09-18T01:14:29.833 INFO:teuthology.orchestra.run.smithi066.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-18T01:14:29.841 INFO:teuthology.orchestra.run.smithi115.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 587 kB/s | 20 kB 00:00 2024-09-18T01:14:29.858 INFO:teuthology.orchestra.run.smithi115.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-18T01:14:29.883 INFO:teuthology.orchestra.run.smithi066.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 709 kB/s | 35 kB 00:00 2024-09-18T01:14:29.892 INFO:teuthology.orchestra.run.smithi115.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-18T01:14:29.917 INFO:teuthology.orchestra.run.smithi066.stdout:(14/37): python3-requests-2.25.1-8.el9.noarch.r 182 kB/s | 125 kB 00:00 2024-09-18T01:14:29.925 INFO:teuthology.orchestra.run.smithi115.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-18T01:14:29.942 INFO:teuthology.orchestra.run.smithi066.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-18T01:14:29.951 INFO:teuthology.orchestra.run.smithi115.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-09-18T01:14:29.976 INFO:teuthology.orchestra.run.smithi115.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-18T01:14:29.984 INFO:teuthology.orchestra.run.smithi066.stdout:(16/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-18T01:14:30.001 INFO:teuthology.orchestra.run.smithi115.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-09-18T01:14:30.009 INFO:teuthology.orchestra.run.smithi066.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 557 kB/s | 51 kB 00:00 2024-09-18T01:14:30.035 INFO:teuthology.orchestra.run.smithi115.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-09-18T01:14:30.060 INFO:teuthology.orchestra.run.smithi115.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-18T01:14:30.076 INFO:teuthology.orchestra.run.smithi066.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-18T01:14:30.086 INFO:teuthology.orchestra.run.smithi115.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-09-18T01:14:30.097 INFO:teuthology.orchestra.run.smithi066.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 199 kB/s | 22 kB 00:00 2024-09-18T01:14:30.113 INFO:teuthology.orchestra.run.smithi066.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.1 MB/s | 43 kB 00:00 2024-09-18T01:14:30.119 INFO:teuthology.orchestra.run.smithi115.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.5 MB/s | 427 kB 00:00 2024-09-18T01:14:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:30.123 INFO:teuthology.orchestra.run.smithi115.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-18T01:14:30.172 INFO:teuthology.orchestra.run.smithi066.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 6.9 MB/s | 358 kB 00:00 2024-09-18T01:14:30.197 INFO:teuthology.orchestra.run.smithi066.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-18T01:14:30.214 INFO:teuthology.orchestra.run.smithi066.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-18T01:14:30.231 INFO:teuthology.orchestra.run.smithi066.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 529 kB/s | 18 kB 00:00 2024-09-18T01:14:30.248 INFO:teuthology.orchestra.run.smithi066.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 690 kB/s | 23 kB 00:00 2024-09-18T01:14:30.265 INFO:teuthology.orchestra.run.smithi066.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 582 kB/s | 19 kB 00:00 2024-09-18T01:14:30.280 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:14:30.282 INFO:teuthology.orchestra.run.smithi066.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 587 kB/s | 20 kB 00:00 2024-09-18T01:14:30.307 INFO:teuthology.orchestra.run.smithi066.stdout:(28/37): python3-urllib3-1.26.5-6.el9.noarch.rp 274 kB/s | 215 kB 00:00 2024-09-18T01:14:30.320 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:14:30.320 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:14:30.324 INFO:teuthology.orchestra.run.smithi066.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 788 kB/s | 46 kB 00:00 2024-09-18T01:14:30.366 INFO:teuthology.orchestra.run.smithi066.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-09-18T01:14:30.383 INFO:teuthology.orchestra.run.smithi066.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 324 kB/s | 16 kB 00:00 2024-09-18T01:14:30.408 INFO:teuthology.orchestra.run.smithi066.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-18T01:14:30.434 INFO:teuthology.orchestra.run.smithi066.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-18T01:14:30.459 INFO:teuthology.orchestra.run.smithi066.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-18T01:14:30.476 INFO:teuthology.orchestra.run.smithi066.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-18T01:14:30.509 INFO:teuthology.orchestra.run.smithi066.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-18T01:14:30.552 INFO:teuthology.orchestra.run.smithi066.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-18T01:14:30.557 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:30.557 INFO:teuthology.orchestra.run.smithi066.stdout:Total 8.1 MB/s | 30 MB 00:03 2024-09-18T01:14:30.718 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:14:30.764 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:14:30.764 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:14:30.770 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:14:30.770 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:14:31.253 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:14:31.253 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:14:32.131 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:14:32.197 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-18T01:14:32.230 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-18T01:14:32.274 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-18T01:14:32.345 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-18T01:14:32.401 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-18T01:14:32.439 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-18T01:14:32.533 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-18T01:14:32.618 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-18T01:14:32.659 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-18T01:14:32.694 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-18T01:14:32.726 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-18T01:14:32.758 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:14:32.760 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-18T01:14:32.794 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-18T01:14:32.804 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-18T01:14:32.834 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-18T01:14:32.842 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-18T01:14:32.869 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-18T01:14:32.882 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-18T01:14:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-18T01:14:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-18T01:14:32.958 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-18T01:14:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-18T01:14:33.006 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-18T01:14:33.039 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-18T01:14:33.065 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-18T01:14:33.136 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-18T01:14:33.213 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-18T01:14:33.214 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-18T01:14:33.293 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-18T01:14:33.345 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-18T01:14:33.384 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-18T01:14:33.384 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-18T01:14:33.418 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-18T01:14:33.433 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-18T01:14:33.448 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-18T01:14:33.482 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-18T01:14:33.488 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-18T01:14:33.516 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-18T01:14:33.565 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-18T01:14:33.609 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-18T01:14:33.642 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-18T01:14:33.684 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-18T01:14:33.691 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-18T01:14:33.730 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-18T01:14:33.731 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-18T01:14:33.808 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-18T01:14:33.808 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-18T01:14:33.897 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-18T01:14:33.939 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-18T01:14:33.960 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-18T01:14:34.050 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-18T01:14:34.053 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-18T01:14:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.n 32/37 2024-09-18T01:14:34.226 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-18T01:14:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 33/37 2024-09-18T01:14:34.264 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-18T01:14:34.275 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 33/37 2024-09-18T01:14:34.275 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:34.275 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T01:14:34.275 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T01:14:34.275 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T01:14:34.275 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:34.322 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-18T01:14:34.386 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-18T01:14:34.581 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-18T01:14:34.634 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-18T01:14:34.714 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-18T01:14:34.806 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-18T01:14:34.966 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-18T01:14:35.090 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-18T01:14:35.261 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.n 32/37 2024-09-18T01:14:35.292 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 33/37 2024-09-18T01:14:35.320 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 33/37 2024-09-18T01:14:35.320 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:35.321 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T01:14:35.321 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T01:14:35.321 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T01:14:35.321 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:35.724 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 34/37 2024-09-18T01:14:35.752 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 34/37 2024-09-18T01:14:35.752 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:35.752 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T01:14:35.752 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T01:14:35.752 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T01:14:35.752 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:36.200 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 35/37 2024-09-18T01:14:36.223 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 35/37 2024-09-18T01:14:36.223 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:36.223 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T01:14:36.223 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T01:14:36.223 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T01:14:36.223 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:36.450 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 36/37 2024-09-18T01:14:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 36/37 2024-09-18T01:14:36.476 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:36.476 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T01:14:36.476 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T01:14:36.476 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T01:14:36.476 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:36.539 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 37/37 2024-09-18T01:14:36.843 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 34/37 2024-09-18T01:14:36.872 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 34/37 2024-09-18T01:14:36.872 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:36.872 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T01:14:36.872 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T01:14:36.872 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T01:14:36.872 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:37.327 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 35/37 2024-09-18T01:14:37.353 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 35/37 2024-09-18T01:14:37.353 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:37.353 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T01:14:37.353 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T01:14:37.353 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T01:14:37.353 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:37.586 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 36/37 2024-09-18T01:14:37.611 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 36/37 2024-09-18T01:14:37.611 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:37.611 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T01:14:37.612 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T01:14:37.612 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T01:14:37.612 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:37.678 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 37/37 2024-09-18T01:14:38.365 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 37/37 2024-09-18T01:14:38.365 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 1/37 2024-09-18T01:14:38.365 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2/37 2024-09-18T01:14:38.365 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 3/37 2024-09-18T01:14:38.365 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 4/37 2024-09-18T01:14:38.365 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 5/37 2024-09-18T01:14:38.365 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.n 6/37 2024-09-18T01:14:38.365 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-18T01:14:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-18T01:14:38.367 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-18T01:14:38.368 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-18T01:14:38.368 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-18T01:14:38.368 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:39.504 INFO:teuthology.orchestra.run.smithi115.stdout: lua-5.4.4-4.el9.x86_64 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T01:14:39.505 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T01:14:39.506 INFO:teuthology.orchestra.run.smithi115.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T01:14:39.507 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T01:14:39.507 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T01:14:39.507 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T01:14:39.507 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:39.507 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 37/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 1/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 3/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 4/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 5/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.n 6/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-18T01:14:39.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-18T01:14:39.540 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-18T01:14:39.541 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-18T01:14:39.541 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-18T01:14:39.541 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-18T01:14:39.541 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-18T01:14:39.541 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-18T01:14:39.542 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-18T01:14:39.542 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-18T01:14:39.542 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-18T01:14:39.542 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-18T01:14:39.542 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-18T01:14:39.542 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-18T01:14:39.543 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-18T01:14:39.543 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-18T01:14:39.900 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-base 2024-09-18T01:14:40.481 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:53 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: lua-5.4.4-4.el9.x86_64 2024-09-18T01:14:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T01:14:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T01:14:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T01:14:40.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T01:14:40.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T01:14:40.568 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:40.568 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:14:40.570 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:14:40.612 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:14:40.613 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:14:40.613 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:14:40.680 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install cephadm 2024-09-18T01:14:40.937 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-base 2024-09-18T01:14:41.257 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:14:41.388 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:14:41.388 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 765 k 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 765 k 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 771 k 2024-09-18T01:14:41.389 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:14:41.569 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:55 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:14:41.650 INFO:teuthology.orchestra.run.smithi115.stdout:cephadm-19.1.1-330.gc2ddf972.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-18T01:14:41.651 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:41.651 INFO:teuthology.orchestra.run.smithi115.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-18T01:14:41.651 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:14:41.656 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:14:41.656 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:14:41.660 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:14:41.662 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:14:41.662 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:14:41.704 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:14:41.705 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:14:41.705 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:14:41.733 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:14:41.783 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install cephadm 2024-09-18T01:14:42.204 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:14:42.294 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:14:42.384 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:56 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:14:42.521 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:14:42.522 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:42.522 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:14:42.522 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:42.522 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:14:42.522 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 765 k 2024-09-18T01:14:42.522 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:42.523 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:14:42.523 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:42.523 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-09-18T01:14:42.523 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:42.523 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 765 k 2024-09-18T01:14:42.523 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 771 k 2024-09-18T01:14:42.523 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:14:42.646 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:14:42.793 INFO:teuthology.orchestra.run.smithi066.stdout:cephadm-19.1.1-330.gc2ddf972.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-18T01:14:42.793 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:42.793 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-18T01:14:42.794 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:14:42.799 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:14:42.799 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:14:42.806 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:14:42.806 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:14:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:14:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:42.876 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:14:42.877 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:14:42.877 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:42.877 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:14:42.878 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:14:43.069 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T01:14:43.346 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:14:43.440 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:14:43.664 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:56 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:14:43.797 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 145 k 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-09-18T01:14:43.798 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:43.799 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 145 k 2024-09-18T01:14:43.799 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 435 k 2024-09-18T01:14:43.799 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:14:43.823 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:14:44.017 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-immutable-object-cache-19.1.1-330.gc2ddf97 666 kB/s | 145 kB 00:00 2024-09-18T01:14:44.017 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:44.018 INFO:teuthology.orchestra.run.smithi115.stdout:Total 662 kB/s | 145 kB 00:00 2024-09-18T01:14:44.018 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:14:44.025 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:14:44.025 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:14:44.038 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:14:44.038 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:44.038 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:14:44.038 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:14:44.038 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:44.038 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:14:44.083 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:14:44.083 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:14:44.239 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T01:14:44.240 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:14:44.318 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-18T01:14:44.342 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-18T01:14:44.342 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:44.342 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T01:14:44.342 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:44.851 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:58 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:14:44.990 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:14:44.990 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-09-18T01:14:44.991 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:44.991 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:14:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 145 k 2024-09-18T01:14:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:44.991 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:14:44.991 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:44.991 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-09-18T01:14:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:44.992 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 145 k 2024-09-18T01:14:44.992 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 435 k 2024-09-18T01:14:44.992 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:14:45.198 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-immutable-object-cache-19.1.1-330.gc2ddf97 707 kB/s | 145 kB 00:00 2024-09-18T01:14:45.198 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:45.198 INFO:teuthology.orchestra.run.smithi066.stdout:Total 702 kB/s | 145 kB 00:00 2024-09-18T01:14:45.199 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:14:45.207 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:14:45.207 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:14:45.256 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-18T01:14:45.256 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:45.256 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:14:45.256 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:45.257 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:45.257 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:14:45.267 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:14:45.267 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:14:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:14:45.500 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-18T01:14:45.502 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr 2024-09-18T01:14:45.525 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-18T01:14:45.525 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:14:45.526 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T01:14:45.526 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:46.093 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:59 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:14:46.183 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:14:46.224 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:14:46.225 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:14:46.225 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:14:46.300 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T01:14:46.487 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-18T01:14:46.487 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:46.488 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:14:46.488 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:14:46.488 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:46.488 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:14:46.702 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr 2024-09-18T01:14:46.873 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:59 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:14:47.004 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 3.6 M 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 32 k 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 16 k 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-18T01:14:47.005 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout:Install 12 Packages 2024-09-18T01:14:47.006 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:14:47.007 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 8.7 M 2024-09-18T01:14:47.007 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 104 M 2024-09-18T01:14:47.007 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:14:47.309 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:01 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:14:47.401 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:14:47.445 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:14:47.446 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:14:47.446 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:14:47.516 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T01:14:48.118 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:02 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:14:48.256 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:14:48.257 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 3.6 M 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 32 k 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 16 k 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-18T01:14:48.258 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-18T01:14:48.259 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-18T01:14:48.259 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T01:14:48.259 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T01:14:48.259 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:48.259 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:14:48.259 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:14:48.259 INFO:teuthology.orchestra.run.smithi066.stdout:Install 12 Packages 2024-09-18T01:14:48.259 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:48.260 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 8.7 M 2024-09-18T01:14:48.260 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 104 M 2024-09-18T01:14:48.260 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:14:48.800 INFO:teuthology.orchestra.run.smithi066.stdout:(1/12): ceph-grafana-dashboards-19.1.1-330.gc2d 217 kB/s | 32 kB 00:00 2024-09-18T01:14:48.825 INFO:teuthology.orchestra.run.smithi066.stdout:(2/12): ceph-prometheus-alerts-19.1.1-330.gc2dd 96 kB/s | 16 kB 00:00 2024-09-18T01:14:48.992 INFO:teuthology.orchestra.run.smithi066.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-09-18T01:14:49.059 INFO:teuthology.orchestra.run.smithi066.stdout:(4/12): ceph-mgr-dashboard-19.1.1-330.gc2ddf972 8.8 MB/s | 3.6 MB 00:00 2024-09-18T01:14:49.134 INFO:teuthology.orchestra.run.smithi066.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.2 MB/s | 548 kB 00:00 2024-09-18T01:14:49.159 INFO:teuthology.orchestra.run.smithi066.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-18T01:14:49.193 INFO:teuthology.orchestra.run.smithi066.stdout:(7/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-18T01:14:49.252 INFO:teuthology.orchestra.run.smithi066.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-18T01:14:49.269 INFO:teuthology.orchestra.run.smithi066.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-18T01:14:49.319 INFO:teuthology.orchestra.run.smithi066.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.3 MB/s | 188 kB 00:00 2024-09-18T01:14:49.369 INFO:teuthology.orchestra.run.smithi066.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.7 MB/s | 2.0 MB 00:00 2024-09-18T01:14:49.969 INFO:teuthology.orchestra.run.smithi066.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 884 kB/s | 863 kB 00:00 2024-09-18T01:14:49.975 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:49.975 INFO:teuthology.orchestra.run.smithi066.stdout:Total 5.1 MB/s | 8.7 MB 00:01 2024-09-18T01:14:50.030 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:14:50.049 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:14:50.050 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:14:50.255 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:14:50.255 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:14:50.798 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:14:50.879 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-18T01:14:50.923 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-18T01:14:50.970 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-18T01:14:50.999 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-18T01:14:51.107 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-18T01:14:51.239 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-18T01:14:51.330 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-18T01:14:51.533 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-18T01:14:51.590 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-18T01:14:51.622 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9. 10/12 2024-09-18T01:14:52.759 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9 11/12 2024-09-18T01:14:52.789 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 12/12 2024-09-18T01:14:53.969 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 12/12 2024-09-18T01:14:53.969 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9 1/12 2024-09-18T01:14:53.969 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 2/12 2024-09-18T01:14:53.969 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9. 3/12 2024-09-18T01:14:53.969 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-18T01:14:53.970 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-18T01:14:53.970 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-18T01:14:53.970 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-18T01:14:53.970 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-18T01:14:53.970 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-18T01:14:53.970 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-18T01:14:53.970 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-18T01:14:54.312 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-18T01:14:54.312 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T01:14:54.313 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T01:14:54.314 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:54.314 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:14:54.556 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T01:14:55.163 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:09 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:14:55.302 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:14:55.303 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-09-18T01:14:55.303 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:14:55.303 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-09-18T01:14:55.303 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:14:55.303 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 7.4 M 2024-09-18T01:14:55.303 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:14:55.303 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T01:14:55.303 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:14:55.304 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-09-18T01:14:55.305 INFO:teuthology.orchestra.run.smithi066.stdout:Install 12 Packages 2024-09-18T01:14:55.305 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:14:55.305 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 38 M 2024-09-18T01:14:55.305 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 212 M 2024-09-18T01:14:55.305 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:14:55.585 INFO:teuthology.orchestra.run.smithi066.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-18T01:14:55.626 INFO:teuthology.orchestra.run.smithi066.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 5.0 MB/s | 801 kB 00:00 2024-09-18T01:14:55.668 INFO:teuthology.orchestra.run.smithi066.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-18T01:14:55.710 INFO:teuthology.orchestra.run.smithi066.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-18T01:14:55.769 INFO:teuthology.orchestra.run.smithi066.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 659 kB/s | 38 kB 00:00 2024-09-18T01:14:55.894 INFO:teuthology.orchestra.run.smithi066.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-09-18T01:14:55.936 INFO:teuthology.orchestra.run.smithi115.stdout:(1/12): ceph-prometheus-alerts-19.1.1-330.gc2dd 64 kB/s | 16 kB 00:00 2024-09-18T01:14:55.961 INFO:teuthology.orchestra.run.smithi115.stdout:(2/12): ceph-grafana-dashboards-19.1.1-330.gc2d 113 kB/s | 32 kB 00:00 2024-09-18T01:14:55.995 INFO:teuthology.orchestra.run.smithi066.stdout:(7/12): ceph-mgr-diskprediction-local-19.1.1-33 14 MB/s | 7.4 MB 00:00 2024-09-18T01:14:56.020 INFO:teuthology.orchestra.run.smithi066.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-18T01:14:56.062 INFO:teuthology.orchestra.run.smithi066.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-18T01:14:56.137 INFO:teuthology.orchestra.run.smithi066.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 13 MB/s | 4.9 MB 00:00 2024-09-18T01:14:56.162 INFO:teuthology.orchestra.run.smithi115.stdout:(3/12): ceph-mgr-dashboard-19.1.1-330.gc2ddf972 7.4 MB/s | 3.6 MB 00:00 2024-09-18T01:14:56.220 INFO:teuthology.orchestra.run.smithi115.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-09-18T01:14:56.287 INFO:teuthology.orchestra.run.smithi115.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-09-18T01:14:56.329 INFO:teuthology.orchestra.run.smithi115.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-18T01:14:56.354 INFO:teuthology.orchestra.run.smithi115.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-09-18T01:14:56.387 INFO:teuthology.orchestra.run.smithi066.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-09-18T01:14:56.388 INFO:teuthology.orchestra.run.smithi115.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.4 MB/s | 144 kB 00:00 2024-09-18T01:14:56.413 INFO:teuthology.orchestra.run.smithi115.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-18T01:14:56.447 INFO:teuthology.orchestra.run.smithi115.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-18T01:14:56.462 INFO:teuthology.orchestra.run.smithi066.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-18T01:14:56.463 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:56.464 INFO:teuthology.orchestra.run.smithi066.stdout:Total 33 MB/s | 38 MB 00:01 2024-09-18T01:14:56.497 INFO:teuthology.orchestra.run.smithi115.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.5 MB/s | 863 kB 00:00 2024-09-18T01:14:56.564 INFO:teuthology.orchestra.run.smithi115.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.7 MB/s | 2.0 MB 00:00 2024-09-18T01:14:56.568 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:14:56.568 INFO:teuthology.orchestra.run.smithi115.stdout:Total 928 kB/s | 8.7 MB 00:09 2024-09-18T01:14:56.619 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:14:56.638 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:14:56.638 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:14:56.831 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:14:56.836 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:14:56.837 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:14:56.859 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:14:56.859 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:14:57.325 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:14:57.325 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:14:57.380 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:14:57.464 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-18T01:14:57.508 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-18T01:14:57.554 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-18T01:14:57.593 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-18T01:14:57.698 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-18T01:14:57.827 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-18T01:14:57.921 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-18T01:14:57.974 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-18T01:14:58.004 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-18T01:14:58.033 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9. 10/12 2024-09-18T01:14:58.139 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:14:58.202 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T01:14:58.244 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T01:14:58.349 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T01:14:58.387 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T01:14:58.572 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T01:14:58.611 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T01:14:58.700 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T01:14:59.082 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9 11/12 2024-09-18T01:14:59.134 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 12/12 2024-09-18T01:14:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T01:14:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T01:15:00.514 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 12/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9 1/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 2/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9. 3/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-18T01:15:00.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-18T01:15:00.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-18T01:15:00.836 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:00.836 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:00.836 INFO:teuthology.orchestra.run.smithi115.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-18T01:15:00.836 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:00.837 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T01:15:01.085 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T01:15:01.663 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:14 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:01.796 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:01.797 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================== 2024-09-18T01:15:01.797 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:15:01.797 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================== 2024-09-18T01:15:01.797 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:01.797 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 7.4 M 2024-09-18T01:15:01.797 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:15:01.797 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T01:15:01.797 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================== 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout:Install 12 Packages 2024-09-18T01:15:01.798 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:01.799 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 38 M 2024-09-18T01:15:01.799 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 212 M 2024-09-18T01:15:01.799 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:01.925 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T01:15:01.953 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 12/12 2024-09-18T01:15:02.239 INFO:teuthology.orchestra.run.smithi115.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-18T01:15:02.281 INFO:teuthology.orchestra.run.smithi115.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-18T01:15:02.373 INFO:teuthology.orchestra.run.smithi115.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-18T01:15:02.423 INFO:teuthology.orchestra.run.smithi115.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-18T01:15:02.498 INFO:teuthology.orchestra.run.smithi115.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 515 kB/s | 38 kB 00:00 2024-09-18T01:15:02.607 INFO:teuthology.orchestra.run.smithi115.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-33 15 MB/s | 7.4 MB 00:00 2024-09-18T01:15:02.774 INFO:teuthology.orchestra.run.smithi115.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-18T01:15:03.149 INFO:teuthology.orchestra.run.smithi115.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.4 MB/s | 3.0 MB 00:00 2024-09-18T01:15:03.250 INFO:teuthology.orchestra.run.smithi115.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.5 MB/s | 250 kB 00:00 2024-09-18T01:15:03.600 INFO:teuthology.orchestra.run.smithi115.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.5 MB/s | 4.9 MB 00:01 2024-09-18T01:15:03.883 INFO:teuthology.orchestra.run.smithi115.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.6 MB/s | 5.1 MB 00:01 2024-09-18T01:15:04.494 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 12/12 2024-09-18T01:15:04.494 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 1/12 2024-09-18T01:15:04.494 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T01:15:04.494 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T01:15:04.494 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T01:15:04.494 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T01:15:04.495 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T01:15:04.495 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T01:15:04.495 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T01:15:04.495 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T01:15:04.495 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T01:15:04.495 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T01:15:04.879 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T01:15:04.880 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T01:15:04.880 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T01:15:04.880 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T01:15:04.880 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T01:15:04.880 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T01:15:04.880 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:04.880 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:05.274 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-rook 2024-09-18T01:15:05.733 INFO:teuthology.orchestra.run.smithi115.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.5 MB/s | 16 MB 00:02 2024-09-18T01:15:05.734 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:05.735 INFO:teuthology.orchestra.run.smithi115.stdout:Total 9.7 MB/s | 38 MB 00:03 2024-09-18T01:15:05.883 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:19 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:06.020 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:06.021 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:15:06.021 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:06.021 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:06.021 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 49 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T01:15:06.022 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T01:15:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T01:15:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T01:15:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:06.023 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:06.023 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:06.023 INFO:teuthology.orchestra.run.smithi066.stdout:Install 13 Packages 2024-09-18T01:15:06.023 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:06.024 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 2.2 M 2024-09-18T01:15:06.024 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 26 M 2024-09-18T01:15:06.024 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:06.090 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:06.116 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:06.117 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:06.394 INFO:teuthology.orchestra.run.smithi066.stdout:(1/13): ceph-mgr-rook-19.1.1-330.gc2ddf972.el9. 290 kB/s | 49 kB 00:00 2024-09-18T01:15:06.411 INFO:teuthology.orchestra.run.smithi066.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 137 kB/s | 26 kB 00:00 2024-09-18T01:15:06.428 INFO:teuthology.orchestra.run.smithi066.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 94 kB/s | 19 kB 00:00 2024-09-18T01:15:06.556 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:06.556 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:06.686 INFO:teuthology.orchestra.run.smithi066.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 615 kB/s | 159 kB 00:00 2024-09-18T01:15:06.736 INFO:teuthology.orchestra.run.smithi066.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 649 kB/s | 222 kB 00:00 2024-09-18T01:15:06.778 INFO:teuthology.orchestra.run.smithi066.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 773 kB/s | 32 kB 00:00 2024-09-18T01:15:06.795 INFO:teuthology.orchestra.run.smithi066.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 849 kB/s | 14 kB 00:00 2024-09-18T01:15:06.812 INFO:teuthology.orchestra.run.smithi066.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 426 kB/s | 54 kB 00:00 2024-09-18T01:15:06.838 INFO:teuthology.orchestra.run.smithi066.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 680 kB/s | 279 kB 00:00 2024-09-18T01:15:06.880 INFO:teuthology.orchestra.run.smithi066.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-18T01:15:06.913 INFO:teuthology.orchestra.run.smithi066.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.0 MB/s | 90 kB 00:00 2024-09-18T01:15:06.930 INFO:teuthology.orchestra.run.smithi066.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 699 kB/s | 59 kB 00:00 2024-09-18T01:15:06.997 INFO:teuthology.orchestra.run.smithi066.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-09-18T01:15:07.000 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:07.000 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-18T01:15:07.067 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:07.083 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:07.083 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:07.181 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:07.183 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:07.330 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:07.414 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T01:15:07.457 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T01:15:07.564 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T01:15:07.617 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T01:15:07.706 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:07.799 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T01:15:07.831 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T01:15:07.889 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T01:15:07.908 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T01:15:07.950 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T01:15:08.014 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T01:15:08.049 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T01:15:08.092 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T01:15:08.207 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T01:15:08.317 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T01:15:08.377 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T01:15:08.560 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T01:15:08.678 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T01:15:09.157 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T01:15:09.198 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T01:15:09.231 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T01:15:09.281 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T01:15:09.309 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 13/13 2024-09-18T01:15:09.905 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T01:15:10.104 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 13/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 1/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T01:15:10.105 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T01:15:10.443 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T01:15:10.443 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:10.443 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:10.443 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:10.444 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T01:15:10.444 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T01:15:10.444 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T01:15:10.444 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T01:15:10.444 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T01:15:10.445 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T01:15:10.445 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T01:15:10.445 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T01:15:10.445 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T01:15:10.445 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T01:15:10.445 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T01:15:10.446 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T01:15:10.446 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:10.446 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:10.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T01:15:10.967 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T01:15:10.989 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 12/12 2024-09-18T01:15:11.312 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:25 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:11.451 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:11.452 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:11.452 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:15:11.452 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:11.452 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:11.452 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 146 k 2024-09-18T01:15:11.452 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout:Install 6 Packages 2024-09-18T01:15:11.453 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:11.454 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 7.0 M 2024-09-18T01:15:11.454 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 33 M 2024-09-18T01:15:11.454 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:11.890 INFO:teuthology.orchestra.run.smithi066.stdout:(1/6): ceph-mgr-cephadm-19.1.1-330.gc2ddf972.el 680 kB/s | 146 kB 00:00 2024-09-18T01:15:11.915 INFO:teuthology.orchestra.run.smithi066.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-09-18T01:15:11.974 INFO:teuthology.orchestra.run.smithi066.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-18T01:15:11.999 INFO:teuthology.orchestra.run.smithi066.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-18T01:15:12.033 INFO:teuthology.orchestra.run.smithi066.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-09-18T01:15:13.556 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 12/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 1/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T01:15:13.557 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T01:15:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T01:15:13.558 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T01:15:13.850 INFO:teuthology.orchestra.run.smithi066.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.7 MB/s | 6.0 MB 00:02 2024-09-18T01:15:13.852 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:13.852 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.9 MB/s | 7.0 MB 00:02 2024-09-18T01:15:13.928 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:13.940 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:13.940 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:13.952 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T01:15:13.953 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:13.953 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:13.953 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:13.953 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T01:15:13.953 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T01:15:13.953 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T01:15:13.954 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T01:15:13.954 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T01:15:13.954 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T01:15:13.954 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T01:15:13.954 INFO:teuthology.orchestra.run.smithi115.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T01:15:13.954 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T01:15:13.954 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T01:15:13.955 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T01:15:13.955 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:13.955 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:14.046 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:14.047 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:14.352 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-rook 2024-09-18T01:15:14.367 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T01:15:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T01:15:14.942 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:27 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:15.073 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:15.074 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:15.074 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:15:15.074 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:15.075 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:15.075 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 49 k 2024-09-18T01:15:15.075 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:15:15.075 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T01:15:15.075 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T01:15:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T01:15:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T01:15:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T01:15:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T01:15:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T01:15:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T01:15:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T01:15:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T01:15:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T01:15:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T01:15:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:15.077 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:15.078 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:15.078 INFO:teuthology.orchestra.run.smithi115.stdout:Install 13 Packages 2024-09-18T01:15:15.078 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:15.079 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 2.2 M 2024-09-18T01:15:15.079 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 26 M 2024-09-18T01:15:15.079 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:15.149 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T01:15:15.240 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T01:15:15.320 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T01:15:15.347 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 6/6 2024-09-18T01:15:15.448 INFO:teuthology.orchestra.run.smithi115.stdout:(1/13): ceph-mgr-rook-19.1.1-330.gc2ddf972.el9. 293 kB/s | 49 kB 00:00 2024-09-18T01:15:15.789 INFO:teuthology.orchestra.run.smithi115.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 38 kB/s | 19 kB 00:00 2024-09-18T01:15:15.881 INFO:teuthology.orchestra.run.smithi115.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 43 kB/s | 26 kB 00:00 2024-09-18T01:15:16.088 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 6/6 2024-09-18T01:15:16.089 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/6 2024-09-18T01:15:16.089 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T01:15:16.089 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T01:15:16.089 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T01:15:16.089 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T01:15:16.190 INFO:teuthology.orchestra.run.smithi115.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 299 kB/s | 222 kB 00:00 2024-09-18T01:15:16.363 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T01:15:16.363 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:16.363 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:16.364 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:16.364 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T01:15:16.364 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T01:15:16.364 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T01:15:16.364 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T01:15:16.364 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T01:15:16.365 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:16.365 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:16.582 INFO:teuthology.orchestra.run.smithi115.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 200 kB/s | 159 kB 00:00 2024-09-18T01:15:16.606 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-fuse 2024-09-18T01:15:16.641 INFO:teuthology.orchestra.run.smithi115.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 560 kB/s | 32 kB 00:00 2024-09-18T01:15:16.666 INFO:teuthology.orchestra.run.smithi115.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 564 kB/s | 14 kB 00:00 2024-09-18T01:15:16.691 INFO:teuthology.orchestra.run.smithi115.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 344 kB/s | 279 kB 00:00 2024-09-18T01:15:16.742 INFO:teuthology.orchestra.run.smithi115.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 2.9 MB/s | 220 kB 00:00 2024-09-18T01:15:16.784 INFO:teuthology.orchestra.run.smithi115.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-18T01:15:16.809 INFO:teuthology.orchestra.run.smithi115.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-18T01:15:16.834 INFO:teuthology.orchestra.run.smithi115.stdout:(12/13): python3-requests-oauthlib-1.3.0-12.el9 83 kB/s | 54 kB 00:00 2024-09-18T01:15:16.942 INFO:teuthology.orchestra.run.smithi115.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-09-18T01:15:16.946 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:16.946 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-18T01:15:17.016 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:17.033 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:17.033 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:17.127 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:17.128 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:17.212 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:31 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:17.352 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:17.353 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:17.353 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:15:17.353 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:17.353 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:17.353 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 897 k 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 977 k 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 2.9 M 2024-09-18T01:15:17.354 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:17.525 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 641 kB/s | 80 kB 00:00 2024-09-18T01:15:17.641 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:17.701 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-fuse-19.1.1-330.gc2ddf972.el9.x86_6 2.9 MB/s | 897 kB 00:00 2024-09-18T01:15:17.701 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:17.701 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.7 MB/s | 977 kB 00:00 2024-09-18T01:15:17.714 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:17.737 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:17.737 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:17.798 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T01:15:17.823 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:17.823 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:17.852 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T01:15:17.907 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T01:15:17.944 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T01:15:17.986 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T01:15:18.018 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:18.096 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T01:15:18.171 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T01:15:18.198 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T01:15:18.246 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T01:15:18.253 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2/2 2024-09-18T01:15:18.941 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2/2 2024-09-18T01:15:18.942 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/2 2024-09-18T01:15:18.965 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T01:15:19.010 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T01:15:19.043 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T01:15:19.100 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T01:15:19.121 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 13/13 2024-09-18T01:15:19.216 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T01:15:19.216 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:19.216 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:19.217 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T01:15:19.217 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:19.217 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:19.482 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-volume 2024-09-18T01:15:19.914 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 13/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 1/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T01:15:19.915 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T01:15:20.101 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:34 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:20.240 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 285 k 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-09-18T01:15:20.241 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:20.242 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 285 k 2024-09-18T01:15:20.242 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.3 M 2024-09-18T01:15:20.242 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:20.248 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T01:15:20.248 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:20.248 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T01:15:20.249 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T01:15:20.250 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:20.250 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:20.471 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-volume-19.1.1-330.gc2ddf972.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-18T01:15:20.472 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:20.472 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-18T01:15:20.472 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:20.481 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:20.481 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:20.499 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T01:15:20.507 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:20.507 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:20.757 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:20.832 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:15:20.849 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:15:20.849 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:15:20.850 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T01:15:20.850 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:21.077 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:34 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:21.210 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:21.210 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 146 k 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:21.211 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:21.212 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:21.212 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2024-09-18T01:15:21.212 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:21.212 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 7.0 M 2024-09-18T01:15:21.213 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 33 M 2024-09-18T01:15:21.213 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:21.477 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-18T01:15:21.502 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): ceph-mgr-cephadm-19.1.1-330.gc2ddf972.el 723 kB/s | 146 kB 00:00 2024-09-18T01:15:21.561 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-18T01:15:21.595 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-18T01:15:21.687 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-18T01:15:21.720 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-09-18T01:15:21.723 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:21.723 INFO:teuthology.orchestra.run.smithi115.stdout:Total 14 MB/s | 7.0 MB 00:00 2024-09-18T01:15:21.794 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:21.805 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:21.805 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:21.867 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:15:21.867 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:21.867 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:21.867 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:21.868 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:21.868 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:21.904 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:21.905 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:22.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados-devel 2024-09-18T01:15:22.211 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:22.394 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T01:15:22.439 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T01:15:22.778 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:22.915 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:22.915 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 127 k 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 127 k 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 456 k 2024-09-18T01:15:22.916 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:22.955 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T01:15:23.050 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T01:15:23.122 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T01:15:23.125 INFO:teuthology.orchestra.run.smithi066.stdout:librados-devel-19.1.1-330.gc2ddf972.el9.x86_64. 606 kB/s | 127 kB 00:00 2024-09-18T01:15:23.126 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:23.126 INFO:teuthology.orchestra.run.smithi066.stdout:Total 602 kB/s | 127 kB 00:00 2024-09-18T01:15:23.126 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:23.132 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:23.132 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:23.151 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 6/6 2024-09-18T01:15:23.200 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:23.201 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:23.357 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:23.468 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 6/6 2024-09-18T01:15:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/6 2024-09-18T01:15:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T01:15:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T01:15:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T01:15:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T01:15:23.927 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:24.182 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:24.183 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:24.183 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:24.183 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:24.183 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:24.183 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:24.197 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T01:15:24.197 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:24.197 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:24.197 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:24.197 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T01:15:24.197 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T01:15:24.197 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T01:15:24.197 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T01:15:24.198 INFO:teuthology.orchestra.run.smithi115.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T01:15:24.198 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:24.198 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:24.440 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs2 2024-09-18T01:15:24.443 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-fuse 2024-09-18T01:15:25.016 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:38 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:25.054 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:25.147 INFO:teuthology.orchestra.run.smithi066.stdout:Package libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:25.148 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:25.148 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:25.148 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 897 k 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout:Install 2 Packages 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 977 k 2024-09-18T01:15:25.149 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 2.9 M 2024-09-18T01:15:25.150 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:25.193 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:25.195 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:15:25.195 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:25.272 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs-devel 2024-09-18T01:15:25.705 INFO:teuthology.orchestra.run.smithi115.stdout:(1/2): ceph-fuse-19.1.1-330.gc2ddf972.el9.x86_6 2.9 MB/s | 897 kB 00:00 2024-09-18T01:15:25.880 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:26.018 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 32 k 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:26.019 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-09-18T01:15:26.020 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:26.020 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 32 k 2024-09-18T01:15:26.020 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 142 k 2024-09-18T01:15:26.020 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:26.047 INFO:teuthology.orchestra.run.smithi115.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 125 kB/s | 80 kB 00:00 2024-09-18T01:15:26.048 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:26.048 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.1 MB/s | 977 kB 00:00 2024-09-18T01:15:26.060 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:26.082 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:26.082 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:26.163 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:26.163 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:26.171 INFO:teuthology.orchestra.run.smithi066.stdout:libcephfs-devel-19.1.1-330.gc2ddf972.el9.x86_64 210 kB/s | 32 kB 00:00 2024-09-18T01:15:26.171 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:26.172 INFO:teuthology.orchestra.run.smithi066.stdout:Total 208 kB/s | 32 kB 00:00 2024-09-18T01:15:26.172 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:26.176 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:26.176 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:26.201 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:26.201 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:26.294 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:26.361 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:26.422 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:26.504 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T01:15:26.571 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2/2 2024-09-18T01:15:26.801 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:27.032 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:27.032 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:27.032 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:27.032 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:27.032 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:27.032 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:27.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados2 2024-09-18T01:15:27.263 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2/2 2024-09-18T01:15:27.263 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/2 2024-09-18T01:15:27.532 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T01:15:27.533 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:27.533 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:27.533 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T01:15:27.533 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:27.533 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:27.816 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:41 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:27.819 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-volume 2024-09-18T01:15:27.909 INFO:teuthology.orchestra.run.smithi066.stdout:Package librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:27.955 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:27.956 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:15:27.957 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:28.028 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librbd1 2024-09-18T01:15:28.421 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:41 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:28.555 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 285 k 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-09-18T01:15:28.556 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:28.557 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 285 k 2024-09-18T01:15:28.557 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 1.3 M 2024-09-18T01:15:28.557 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:28.631 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:28.723 INFO:teuthology.orchestra.run.smithi066.stdout:Package librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:28.768 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:28.769 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:15:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:28.794 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-volume-19.1.1-330.gc2ddf972.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-18T01:15:28.795 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:28.795 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-18T01:15:28.795 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:28.803 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:28.803 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:28.827 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:28.827 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:28.841 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rados 2024-09-18T01:15:29.044 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:29.112 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:15:29.128 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:15:29.128 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:15:29.128 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T01:15:29.128 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:29.451 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:29.544 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:29.590 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:29.591 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:15:29.591 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:29.659 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rgw 2024-09-18T01:15:30.149 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-18T01:15:30.149 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:30.149 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:30.149 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-18T01:15:30.149 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:30.149 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:30.275 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:44 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:30.367 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:30.413 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:30.414 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:15:30.415 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:30.433 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados-devel 2024-09-18T01:15:30.482 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-cephfs 2024-09-18T01:15:31.027 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:44 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:31.088 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:45 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:31.159 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:31.160 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:31.160 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-09-18T01:15:31.160 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:31.160 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 127 k 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 127 k 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 456 k 2024-09-18T01:15:31.161 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:31.179 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:31.225 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:31.226 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:15:31.226 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:31.288 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rbd 2024-09-18T01:15:31.358 INFO:teuthology.orchestra.run.smithi115.stdout:librados-devel-19.1.1-330.gc2ddf972.el9.x86_64. 646 kB/s | 127 kB 00:00 2024-09-18T01:15:31.359 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:31.359 INFO:teuthology.orchestra.run.smithi115.stdout:Total 640 kB/s | 127 kB 00:00 2024-09-18T01:15:31.359 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:31.364 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:31.364 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:31.427 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:31.427 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:31.576 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:31.672 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:31.898 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:45 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:31.991 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:32.038 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:32.039 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:15:32.039 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:32.111 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-fuse 2024-09-18T01:15:32.134 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:32.400 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:32.400 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:32.401 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:32.401 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:32.401 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:32.401 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:32.670 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs2 2024-09-18T01:15:32.726 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:46 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:32.868 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 85 k 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-09-18T01:15:32.869 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:32.870 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 85 k 2024-09-18T01:15:32.870 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 231 k 2024-09-18T01:15:32.870 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:33.066 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-fuse-19.1.1-330.gc2ddf972.el9.x86_64.rpm 436 kB/s | 85 kB 00:00 2024-09-18T01:15:33.066 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:33.067 INFO:teuthology.orchestra.run.smithi066.stdout:Total 432 kB/s | 85 kB 00:00 2024-09-18T01:15:33.067 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:33.075 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:33.075 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:33.142 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:33.142 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:33.282 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:46 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:33.309 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:33.374 INFO:teuthology.orchestra.run.smithi115.stdout:Package libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:33.408 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:33.417 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:33.418 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:15:33.418 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:33.488 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs-devel 2024-09-18T01:15:33.810 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:34.049 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:34.049 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:34.049 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:34.049 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:34.049 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:34.050 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:34.065 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:47 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:34.197 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:34.198 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-09-18T01:15:34.198 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:34.198 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 32 k 2024-09-18T01:15:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:34.198 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:34.198 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:34.199 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-09-18T01:15:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:34.199 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 32 k 2024-09-18T01:15:34.199 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 142 k 2024-09-18T01:15:34.199 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:34.311 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-mirror 2024-09-18T01:15:34.354 INFO:teuthology.orchestra.run.smithi115.stdout:libcephfs-devel-19.1.1-330.gc2ddf972.el9.x86_64 205 kB/s | 32 kB 00:00 2024-09-18T01:15:34.355 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:34.355 INFO:teuthology.orchestra.run.smithi115.stdout:Total 203 kB/s | 32 kB 00:00 2024-09-18T01:15:34.355 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:34.358 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:34.358 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:34.379 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:34.379 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:34.468 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:34.568 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:34.925 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:48 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:34.948 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:35.069 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:35.070 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:35.070 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-09-18T01:15:35.070 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:35.070 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:35.070 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.1 M 2024-09-18T01:15:35.070 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:35.070 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:35.070 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:35.071 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-09-18T01:15:35.071 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:35.071 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 3.1 M 2024-09-18T01:15:35.071 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 13 M 2024-09-18T01:15:35.071 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:35.185 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:35.185 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:35.185 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:35.185 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:35.185 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:35.185 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:35.367 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados2 2024-09-18T01:15:35.419 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-mirror-19.1.1-330.gc2ddf972.el9.x86_64.rpm 8.9 MB/s | 3.1 MB 00:00 2024-09-18T01:15:35.419 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:35.420 INFO:teuthology.orchestra.run.smithi066.stdout:Total 8.9 MB/s | 3.1 MB 00:00 2024-09-18T01:15:35.420 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:35.429 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:35.430 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:35.516 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:35.516 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:35.936 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:48 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:35.970 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:36.025 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:36.028 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:36.055 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:36.055 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:15:36.055 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T01:15:36.055 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T01:15:36.055 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T01:15:36.055 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:36.068 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:36.069 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:15:36.069 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:36.130 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librbd1 2024-09-18T01:15:36.704 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:49 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:36.793 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:36.836 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:36.837 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:15:36.837 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:36.896 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rados 2024-09-18T01:15:37.189 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:37.189 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:37.189 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:37.190 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:37.190 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:37.190 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:37.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-nbd 2024-09-18T01:15:37.478 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:50 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:37.568 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:37.610 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:37.611 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:15:37.611 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:37.682 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rgw 2024-09-18T01:15:38.041 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:52 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:38.180 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:38.181 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:38.181 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-09-18T01:15:38.181 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:38.181 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-09-18T01:15:38.181 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 170 k 2024-09-18T01:15:38.181 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:38.182 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-09-18T01:15:38.182 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-09-18T01:15:38.182 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-09-18T01:15:38.182 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:38.182 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 170 k 2024-09-18T01:15:38.182 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 494 k 2024-09-18T01:15:38.182 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-09-18T01:15:38.260 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:51 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:38.349 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:38.391 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:38.392 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:15:38.392 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:38.397 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-nbd-19.1.1-330.gc2ddf972.el9.x86_64.rpm 795 kB/s | 170 kB 00:00 2024-09-18T01:15:38.397 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:38.398 INFO:teuthology.orchestra.run.smithi066.stdout:Total 789 kB/s | 170 kB 00:00 2024-09-18T01:15:38.398 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-09-18T01:15:38.406 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-09-18T01:15:38.406 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-09-18T01:15:38.458 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-cephfs 2024-09-18T01:15:38.474 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-09-18T01:15:38.475 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-09-18T01:15:38.635 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-09-18T01:15:38.730 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:39.034 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:52 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:39.123 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:39.166 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:39.167 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:15:39.167 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:39.177 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:39.230 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rbd 2024-09-18T01:15:39.423 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:39.423 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:39.423 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-09-18T01:15:39.423 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:39.424 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:39.424 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:39.637 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-mirror 2024-09-18T01:15:39.830 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:52 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:39.928 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:39.972 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:39.973 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:15:39.973 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:40.041 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-fuse 2024-09-18T01:15:40.247 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:54 ago on Wed 18 Sep 2024 01:13:46 AM UTC. 2024-09-18T01:15:40.342 INFO:teuthology.orchestra.run.smithi066.stdout:Package rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:40.389 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-09-18T01:15:40.390 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-09-18T01:15:40.390 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-09-18T01:15:40.461 DEBUG:teuthology.parallel:result is None 2024-09-18T01:15:40.614 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:53 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:40.746 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:40.747 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:40.747 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-09-18T01:15:40.748 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:40.748 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:40.748 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 85 k 2024-09-18T01:15:40.748 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:40.748 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:40.749 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:40.749 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-09-18T01:15:40.749 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:40.749 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 85 k 2024-09-18T01:15:40.749 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 231 k 2024-09-18T01:15:40.749 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:40.940 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-fuse-19.1.1-330.gc2ddf972.el9.x86_64.rpm 444 kB/s | 85 kB 00:00 2024-09-18T01:15:40.941 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:40.941 INFO:teuthology.orchestra.run.smithi115.stdout:Total 441 kB/s | 85 kB 00:00 2024-09-18T01:15:40.941 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:40.948 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:41.010 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:41.011 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:41.157 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:41.254 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:41.650 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:41.890 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:41.891 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:41.891 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:41.891 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:41.891 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:41.891 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:42.157 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-mirror 2024-09-18T01:15:42.749 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:55 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:42.885 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:42.885 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.1 M 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 3.1 M 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 13 M 2024-09-18T01:15:42.886 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:43.236 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-mirror-19.1.1-330.gc2ddf972.el9.x86_64.rpm 8.9 MB/s | 3.1 MB 00:00 2024-09-18T01:15:43.236 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:43.236 INFO:teuthology.orchestra.run.smithi115.stdout:Total 8.8 MB/s | 3.1 MB 00:00 2024-09-18T01:15:43.237 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:43.246 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:43.246 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:43.328 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:43.329 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:43.759 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:43.819 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:43.845 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:43.845 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T01:15:43.846 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T01:15:43.846 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T01:15:43.846 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T01:15:43.846 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:44.928 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:44.928 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:44.928 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:44.928 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:44.928 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:44.928 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:45.189 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-nbd 2024-09-18T01:15:45.787 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:58 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:45.919 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:45.919 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 170 k 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 170 k 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 494 k 2024-09-18T01:15:45.920 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-18T01:15:46.131 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-nbd-19.1.1-330.gc2ddf972.el9.x86_64.rpm 808 kB/s | 170 kB 00:00 2024-09-18T01:15:46.132 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-18T01:15:46.132 INFO:teuthology.orchestra.run.smithi115.stdout:Total 803 kB/s | 170 kB 00:00 2024-09-18T01:15:46.132 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-18T01:15:46.140 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-18T01:15:46.140 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-18T01:15:46.204 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-18T01:15:46.204 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-18T01:15:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-18T01:15:46.470 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:46.902 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:47.152 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-18T01:15:47.152 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:47.152 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-18T01:15:47.152 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-18T01:15:47.152 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:47.152 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:47.389 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-mirror 2024-09-18T01:15:47.987 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:00 ago on Wed 18 Sep 2024 01:13:47 AM UTC. 2024-09-18T01:15:48.076 INFO:teuthology.orchestra.run.smithi115.stdout:Package rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-18T01:15:48.122 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-18T01:15:48.123 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-09-18T01:15:48.124 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-18T01:15:48.190 DEBUG:teuthology.parallel:result is None 2024-09-18T01:15:48.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-18T01:15:48.334 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T01:15:48.360 INFO:teuthology.orchestra.run.smithi066.stdout:19.1.1-330.gc2ddf972.el9 2024-09-18T01:15:48.361 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-330.gc2ddf972.el9 2024-09-18T01:15:48.361 INFO:teuthology.task.install:The correct ceph version 19.1.1-330.gc2ddf972 is installed. 2024-09-18T01:15:48.363 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-18T01:15:48.502 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T01:15:48.523 INFO:teuthology.orchestra.run.smithi115.stdout:19.1.1-330.gc2ddf972.el9 2024-09-18T01:15:48.524 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-330.gc2ddf972.el9 2024-09-18T01:15:48.524 INFO:teuthology.task.install:The correct ceph version 19.1.1-330.gc2ddf972 is installed. 2024-09-18T01:15:48.527 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T01:15:48.527 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:48.527 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T01:15:48.559 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:15:48.559 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T01:15:48.589 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T01:15:48.590 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:48.590 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T01:15:48.630 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T01:15:48.699 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:15:48.699 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T01:15:48.724 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T01:15:48.789 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T01:15:48.789 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:48.790 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T01:15:48.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T01:15:48.888 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:15:48.888 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T01:15:48.913 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T01:15:48.978 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T01:15:48.978 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:48.979 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T01:15:49.008 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T01:15:49.077 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:15:49.077 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T01:15:49.102 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T01:15:49.167 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T01:15:49.246 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T01:15:49.246 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 777 /var/log/ceph 2024-09-18T01:15:49.249 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /var/log/ceph 2024-09-18T01:15:49.274 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T01:15:49.274 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T01:15:49.317 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T01:15:49.342 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T01:15:49.342 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T01:15:49.385 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T01:15:49.421 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-18T01:15:49.421 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T01:15:49.421 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909706', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1983, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'sleep_before_teardown': 0, 'subset': '79/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7', 'targets': {'smithi066.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGn6aYBEPHizdpZJ+69qOLn/mMp4MZTTPpuLz8kHDglD/5eyJWhiY+WovdoH4DdROl74VGgdBmBtT4aCUx/yNCU=', 'smithi115.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbKvIrtgNKXW7OrMynfsNnBa2DwHxfRxcT/MU/kRBzd2q9W/ve+QPw3O5ahdvSfbu3T2PgayQrYyvuW1+ta3Fc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T01:15:49.422 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-18T01:15:49.453 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-18T01:15:49.467 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T01:15:49.510 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T01:15:49.535 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:49.536 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T01:15:49.592 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T01:15:49.592 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 2024-09-18T01:15:49.649 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T01:15:49.649 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T01:15:49.649 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-18T01:15:49.649 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T01:15:49.649 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T01:15:49.650 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-09-18 01:15:36.330577189 +0000 2024-09-18T01:15:49.650 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-09-18 01:13:06.526217407 +0000 2024-09-18T01:15:49.650 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-09-18 01:13:06.526217407 +0000 2024-09-18T01:15:49.650 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-09-18 01:13:06.526217407 +0000 2024-09-18T01:15:49.650 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T01:15:49.714 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-09-18T01:15:49.714 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-09-18T01:15:49.714 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.0002442 s, 2.1 MB/s 2024-09-18T01:15:49.716 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T01:15:49.772 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 2024-09-18T01:15:49.828 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T01:15:49.828 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T01:15:49.828 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-18T01:15:49.828 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T01:15:49.829 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T01:15:49.829 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-09-18 01:15:36.330577189 +0000 2024-09-18T01:15:49.829 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-09-18 01:13:06.860215362 +0000 2024-09-18T01:15:49.829 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-09-18 01:13:06.860215362 +0000 2024-09-18T01:15:49.829 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-09-18 01:13:06.860215362 +0000 2024-09-18T01:15:49.829 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T01:15:49.892 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-09-18T01:15:49.892 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-09-18T01:15:49.893 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000201026 s, 2.5 MB/s 2024-09-18T01:15:49.894 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T01:15:49.951 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 2024-09-18T01:15:50.007 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T01:15:50.007 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T01:15:50.007 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-18T01:15:50.007 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T01:15:50.007 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T01:15:50.007 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-09-18 01:15:36.330577189 +0000 2024-09-18T01:15:50.007 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-09-18 01:13:07.223213139 +0000 2024-09-18T01:15:50.008 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-09-18 01:13:07.223213139 +0000 2024-09-18T01:15:50.008 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-09-18 01:13:07.223213139 +0000 2024-09-18T01:15:50.008 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T01:15:50.071 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-09-18T01:15:50.071 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-09-18T01:15:50.071 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000198662 s, 2.6 MB/s 2024-09-18T01:15:50.072 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T01:15:50.129 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-09-18 01:15:36.331577181 +0000 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-09-18 01:13:07.551211130 +0000 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-09-18 01:13:07.551211130 +0000 2024-09-18T01:15:50.185 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-09-18 01:13:07.551211130 +0000 2024-09-18T01:15:50.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T01:15:50.249 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-09-18T01:15:50.249 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-09-18T01:15:50.250 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000183638 s, 2.8 MB/s 2024-09-18T01:15:50.251 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T01:15:50.308 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T01:15:50.308 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-18T01:15:50.308 INFO:tasks.ceph:Generating config... 2024-09-18T01:15:50.309 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-18T01:15:50.309 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-18T01:15:50.309 INFO:tasks.ceph:[client] log to stderr = False 2024-09-18T01:15:50.309 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-18T01:15:50.309 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-18T01:15:50.309 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.310 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.310 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.310 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.310 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.310 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.310 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.310 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.310 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T01:15:50.310 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-18T01:15:50.310 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-18T01:15:50.310 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T01:15:50.310 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T01:15:50.310 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T01:15:50.310 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T01:15:50.311 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T01:15:50.311 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-18T01:15:50.312 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T01:15:50.312 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T01:15:50.312 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T01:15:50.312 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T01:15:50.312 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T01:15:50.312 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-18T01:15:50.312 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-18T01:15:50.408 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/cluster1.keyring 2024-09-18T01:15:50.411 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-18T01:15:50.454 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-18T01:15:50.520 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.66')] 2024-09-18T01:15:50.521 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.66', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-18T01:15:50.521 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:50.521 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T01:15:50.576 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.66 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-18T01:15:50.658 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-18T01:15:50.658 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: generated fsid 5799928d-af76-40cf-a91d-c6362506fa2c 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:setting min_mon_release = quincy 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:epoch 0 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:fsid 5799928d-af76-40cf-a91d-c6362506fa2c 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:last_changed 2024-09-18T01:15:50.658648+0000 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:created 2024-09-18T01:15:50.658648+0000 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:min_mon_release 17 (quincy) 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:election_strategy: 1 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:0: [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] mon.a 2024-09-18T01:15:50.659 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-18T01:15:50.660 DEBUG:teuthology.orchestra.run.smithi066:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T01:15:50.716 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 5799928d-af76-40cf-a91d-c6362506fa2c... 2024-09-18T01:15:50.717 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-18T01:15:50.759 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-18T01:15:50.801 INFO:teuthology.orchestra.run.smithi066.stdout:[global] 2024-09-18T01:15:50.801 INFO:teuthology.orchestra.run.smithi066.stdout: chdir = "" 2024-09-18T01:15:50.801 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T01:15:50.801 INFO:teuthology.orchestra.run.smithi066.stdout: auth supported = cephx 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: filestore xattr use omap = true 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: mon clock drift allowed = 1.000 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: osd crush chooseleaf type = 0 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: auth debug = true 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on old message = true 2024-09-18T01:15:50.802 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on bug = true 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: mon pg warn max object skew = 0 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default size = 2 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow primary affinity = true 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow pg remap = true 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on legacy crush tunables = false 2024-09-18T01:15:50.803 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on crush straw calc version zero = false 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on no sortbitwise = false 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on osd down out interval zero = false 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on too few osds = false 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: mon_allow_pool_size_one = true 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: osd default data pool replay window = 5 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.804 INFO:teuthology.orchestra.run.smithi066.stdout: mon allow pool delete = true 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: mon cluster log file level = debug 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: debug asserts on shutdown = true 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: mon health detail to clog = false 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: mon host = 172.21.15.66 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: mon client directed command retry = 5 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on pool no app = False 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: ms inject socket failures = 5000 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: fsid = 5799928d-af76-40cf-a91d-c6362506fa2c 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout:[osd] 2024-09-18T01:15:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: osd journal size = 100 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub load threshold = 5.0 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub max interval = 600 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd mclock profile = high_recovery_ops 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd recover clone overlap = true 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd recovery max chunk = 1048576 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug shutdown = true 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug op order = true 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug verify stray on activate = true 2024-09-18T01:15:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug trim objects = true 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: osd open classes on start = true 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug pg log writeout = true 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: osd deep scrub update digest min age = 30 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: osd map max advance = 10 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: journal zero on create = true 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: filestore ondisk finisher threads = 3 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: filestore apply finisher threads = 3 2024-09-18T01:15:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: bdev debug aio = true 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug misdirected ops = true 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore block size = 96636764160 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore compression algorithm = snappy 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore compression mode = aggressive 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore fsck on mount = True 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: debug bluefs = 1/20 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: debug bluestore = 1/20 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: debug osd = 20 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: debug rocksdb = 4/10 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T01:15:50.808 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd full ratio = 0.9 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd nearfull ratio = 0.8 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: osd failsafe full ratio = 0.95 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: osd objectstore = bluestore 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr] 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: debug mgr = 20 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: mgr/telemetry/nag = false 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.809 INFO:teuthology.orchestra.run.smithi066.stdout:[mon] 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: debug paxos = 20 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: mon data avail warn = 5 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: mon mgr mkfs grace = 240 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd reporter subtree level = osd 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd prime pg temp = true 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2024-09-18T01:15:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: # 1m isn't quite enough 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: mon_down_mkfs_grace = 2m 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_filestore_osds = false 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:15:50.811 INFO:teuthology.orchestra.run.smithi066.stdout:[client] 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: rgw cache enabled = true 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable ops log = true 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable usage log = true 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: debug rbd = 20 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: debug rbd_mirror = 15 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: log to stderr = False 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: rbd default features = 125 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: rbd mirror image policy type = none 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.a] 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.0] 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.812 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.1] 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.2] 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.3] 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.813 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: chdir = "" 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: auth supported = cephx 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: filestore xattr use omap = true 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: mon clock drift allowed = 1.000 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: osd crush chooseleaf type = 0 2024-09-18T01:15:50.814 INFO:teuthology.orchestra.run.smithi115.stdout: auth debug = true 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on old message = true 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on bug = true 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: mon pg warn max object skew = 0 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.815 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default size = 2 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow primary affinity = true 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow pg remap = true 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on legacy crush tunables = false 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on crush straw calc version zero = false 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on no sortbitwise = false 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on osd down out interval zero = false 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on too few osds = false 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: mon_allow_pool_size_one = true 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.816 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: osd default data pool replay window = 5 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: mon allow pool delete = true 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: mon cluster log file level = debug 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: debug asserts on shutdown = true 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: mon health detail to clog = false 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: mon host = 172.21.15.66 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: mon client directed command retry = 5 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on pool no app = False 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: ms inject socket failures = 5000 2024-09-18T01:15:50.817 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = 5799928d-af76-40cf-a91d-c6362506fa2c 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout:[osd] 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd journal size = 100 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub load threshold = 5.0 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub max interval = 600 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock profile = high_recovery_ops 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd recover clone overlap = true 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd recovery max chunk = 1048576 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug shutdown = true 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug op order = true 2024-09-18T01:15:50.818 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify stray on activate = true 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug trim objects = true 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: osd open classes on start = true 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug pg log writeout = true 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: osd deep scrub update digest min age = 30 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: osd map max advance = 10 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: journal zero on create = true 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: filestore ondisk finisher threads = 3 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: filestore apply finisher threads = 3 2024-09-18T01:15:50.819 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: bdev debug aio = true 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug misdirected ops = true 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore block size = 96636764160 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore compression algorithm = snappy 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore compression mode = aggressive 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore fsck on mount = True 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluefs = 1/20 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluestore = 1/20 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: debug osd = 20 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: debug rocksdb = 4/10 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd full ratio = 0.9 2024-09-18T01:15:50.820 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd nearfull ratio = 0.8 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: osd failsafe full ratio = 0.95 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: osd objectstore = bluestore 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr] 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: debug mgr = 20 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: mgr/telemetry/nag = false 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.821 INFO:teuthology.orchestra.run.smithi115.stdout:[mon] 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: debug paxos = 20 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: mon data avail warn = 5 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: mon mgr mkfs grace = 240 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd reporter subtree level = osd 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd prime pg temp = true 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T01:15:50.822 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: # 1m isn't quite enough 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: mon_down_mkfs_grace = 2m 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_filestore_osds = false 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout:[client] 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: rgw cache enabled = true 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable ops log = true 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable usage log = true 2024-09-18T01:15:50.823 INFO:teuthology.orchestra.run.smithi115.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: debug rbd = 20 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: debug rbd_mirror = 15 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: log to stderr = False 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: rbd default features = 125 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: rbd mirror image policy type = none 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.a] 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.0] 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.1] 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.824 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.2] 2024-09-18T01:15:50.825 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.825 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.825 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.3] 2024-09-18T01:15:50.825 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:15:50.825 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:15:50.826 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-18T01:15:50.826 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-18T01:15:50.922 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T01:15:50.922 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:50.922 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-18T01:15:50.938 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:50.938 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-18T01:15:50.994 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:15:50.994 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:50.994 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-18T01:15:50.994 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-18T01:15:51.070 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:15:51.070 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-18T01:15:51.126 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:15:51.126 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:15:51.126 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-18T01:15:51.126 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-18T01:15:51.166 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:15:51.166 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-18T01:15:51.222 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T01:15:51.223 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T01:15:51.223 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-18T01:15:51.275 INFO:teuthology.orchestra.run.smithi066.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-18T01:15:51.277 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T01:15:51.278 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T01:15:51.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-18T01:15:51.320 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-18T01:15:51.333 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-18T01:15:51.388 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-18T01:15:51.401 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-18T01:15:51.443 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-18T01:15:51.456 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-18T01:15:51.536 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-18T01:15:51.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-18T01:15:51.629 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-18T01:15:51.642 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-18T01:15:51.724 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-18T01:15:51.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-18T01:15:51.817 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-18T01:15:51.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-18T01:15:51.910 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-18T01:15:51.926 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-18T01:15:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-18T01:15:52.019 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T01:15:52.019 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-18T01:15:52.019 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-18T01:15:52.049 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T01:15:52.049 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-18T01:15:52.049 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:15:52.049 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T01:15:52.125 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T01:15:52.125 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T01:15:52.125 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T01:15:52.125 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T01:15:52.125 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T01:15:52.125 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-09-18T01:15:52.125 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T01:15:52.125 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T01:15:52.126 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T01:15:52.126 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T01:15:52.631 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-09-18T01:15:52.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-09-18T01:15:52.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-18T01:15:52.763 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-18T01:15:52.798 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-18T01:15:52.867 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T01:15:52.867 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-18T01:15:52.867 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:15:52.867 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T01:15:52.942 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T01:15:52.943 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T01:15:53.394 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-09-18T01:15:53.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-09-18T01:15:53.396 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-18T01:15:53.487 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-18T01:15:53.517 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-18T01:15:53.584 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T01:15:53.584 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-18T01:15:53.585 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:15:53.585 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T01:15:53.658 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T01:15:54.150 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-09-18T01:15:54.152 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-09-18T01:15:54.152 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-18T01:15:54.289 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-18T01:15:54.320 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-18T01:15:54.424 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:15:54.423+0000 7f57c1f2c640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-18T01:15:54.426 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:15:54.424+0000 7f57c1f2c640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-18T01:15:54.426 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:15:54.424+0000 7f57c1f2c640 -1 bdev(0x55d33671b800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-18T01:15:54.426 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:15:54.425+0000 7f57c1f2c640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-18T01:15:57.587 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-18T01:15:57.618 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-18T01:15:57.711 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:15:57.710+0000 7f2848c19640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-18T01:15:57.711 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:15:57.711+0000 7f2848c19640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-18T01:15:57.712 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:15:57.711+0000 7f2848c19640 -1 bdev(0x55c5a0783800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-18T01:15:57.712 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:15:57.711+0000 7f2848c19640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-18T01:16:00.619 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-18T01:16:00.650 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-18T01:16:00.743 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:16:00.742+0000 7f522241c640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-18T01:16:00.743 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:16:00.742+0000 7f522241c640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-18T01:16:00.743 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:16:00.742+0000 7f522241c640 -1 bdev(0x5619a0545800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-18T01:16:00.743 INFO:teuthology.orchestra.run.smithi066.stderr:2024-09-18T01:16:00.742+0000 7f522241c640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-18T01:16:03.669 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-18T01:16:03.700 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T01:16:03.700 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:03.700 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-18T01:16:03.769 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:03.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-18T01:16:03.837 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:03.837 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-18T01:16:03.904 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:03.904 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-18T01:16:03.971 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:03.971 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-18T01:16:04.027 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:04.027 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-18T01:16:04.046 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:04.046 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-18T01:16:04.105 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:04.106 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-18T01:16:04.163 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:04.163 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-18T01:16:04.221 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:04.221 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-18T01:16:04.279 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:04.279 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-18T01:16:04.337 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:04.337 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-18T01:16:04.394 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:04.394 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-18T01:16:04.451 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T01:16:04.451 DEBUG:teuthology.orchestra.run.smithi066:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-18T01:16:04.476 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr.x] 2024-09-18T01:16:04.476 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepm6txpEBAA5YXCdjcHqwE7QzXWHhOS9g== 2024-09-18T01:16:04.476 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.0] 2024-09-18T01:16:04.476 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDKKepmoAJRGRAAzcZtowtaT39n9Pqtj2pFeA== 2024-09-18T01:16:04.476 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.1] 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDNKepmdztnKhAAPqEAXiff5n/kGoSIsMULZg== 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.2] 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDQKepmwmhJLBAAKRdZ52cVTqwcLKyDOjyJzA== 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout:[client.0] 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepmFTgWExAAHjok5CyEXTkCTLurQlMvGw== 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror] 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepm7KQiFxAA1f8smTcAJRy5aU2QQRA+dg== 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.0] 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepm3+JuGhAA4d9ZTJcdVGecnU6J+O3v2A== 2024-09-18T01:16:04.477 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.1] 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepmgj38HxAAbsBzpXOLd0n9uSTUrdicVQ== 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.2] 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepmVeKLJRAAAx1nRW8W0e0s8UqgohXdZw== 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.3] 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepmej8uKxAAJN/ePMQ64SN25RdB064RVQ== 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.4] 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepmhgG4MBAAGBqBNLaInI4Yc6B+HBAAhA== 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.5] 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDHKepm1VhHNhAA7d+0d/8nX8hNGRbYtudoHQ== 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.6] 2024-09-18T01:16:04.478 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQDIKepmTStkABAAhmd/F9YxLCQcn2hvv5mM5w== 2024-09-18T01:16:04.479 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T01:16:04.572 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T01:16:04.633 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T01:16:04.686 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T01:16:04.738 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:04.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:04.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:04.903 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:04.959 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:05.013 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:05.069 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:05.124 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:05.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:05.234 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T01:16:05.234 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-18T01:16:05.259 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-18T01:16:05.356 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-18T01:16:05.382 DEBUG:teuthology.orchestra.run.smithi066:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-18T01:16:05.438 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-18T01:16:05.438 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-18T01:16:05.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-18T01:16:05.480 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-18T01:16:05.481 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-18T01:16:05.481 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-18T01:16:05.481 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-18T01:16:05.483 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-18T01:16:05.483 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T01:16:05.483 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-18T01:16:05.660 INFO:teuthology.orchestra.run.smithi066.stderr:adjusted tunables profile to default 2024-09-18T01:16:05.672 INFO:tasks.ceph:check_enable_crimson: False 2024-09-18T01:16:05.672 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-18T01:16:05.673 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:05.673 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-18T01:16:05.699 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:05.699 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-18T01:16:05.765 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:05.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-18T01:16:05.831 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster cluster1 osd new 81a7cd46-8949-4110-bb1e-add42dfb2660 0 2024-09-18T01:16:05.865 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:05.864+0000 7feae7d2d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T01:16:05.908 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:05.908+0000 7feae7d2d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T01:16:06.046 INFO:teuthology.orchestra.run.smithi066.stdout:0 2024-09-18T01:16:06.058 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster cluster1 osd new 88aec54c-891e-4b4f-967d-f7bb2e7d4c71 1 2024-09-18T01:16:06.062 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:06.061+0000 7feae7d2d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-18T01:16:06.116 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:06.115+0000 7feae7d2d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T01:16:06.240 INFO:teuthology.orchestra.run.smithi066.stdout:1 2024-09-18T01:16:06.251 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster cluster1 osd new 2630c86f-e0ff-400e-93ca-84a26a630e16 2 2024-09-18T01:16:06.292 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:06.292+0000 7feae7d2d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T01:16:06.411 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:06.411+0000 7feae7d2d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T01:16:06.473 INFO:teuthology.orchestra.run.smithi066.stdout:2 2024-09-18T01:16:06.485 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-18T01:16:06.485 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-18T01:16:06.487 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-18T01:16:06.488 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-18T01:16:06.488 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-18T01:16:06.490 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-18T01:16:06.490 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-18T01:16:06.490 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-18T01:16:06.492 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-18T01:16:06.493 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-18T01:16:06.670 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:06.670 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":5,"fsid":"5799928d-af76-40cf-a91d-c6362506fa2c","created":"2024-09-18T01:16:05.571663+0000","modified":"2024-09-18T01:16:06.473310+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T01:16:06.473310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"81a7cd46-8949-4110-bb1e-add42dfb2660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88aec54c-891e-4b4f-967d-f7bb2e7d4c71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2630c86f-e0ff-400e-93ca-84a26a630e16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:06.682 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-18T01:16:06.682 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T01:16:06.763 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:06.763+0000 7feae7d2d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T01:16:06.816 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:06.815+0000 7feae7d2d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T01:16:06.973 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:06.973+0000 7feae7d2d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T01:16:06.984 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-18T01:16:07.151 INFO:teuthology.misc.health.smithi066.stdout: 2024-09-18T01:16:07.151 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":5,"fsid":"5799928d-af76-40cf-a91d-c6362506fa2c","created":"2024-09-18T01:16:05.571663+0000","modified":"2024-09-18T01:16:06.473310+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T01:16:06.473310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"81a7cd46-8949-4110-bb1e-add42dfb2660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88aec54c-891e-4b4f-967d-f7bb2e7d4c71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2630c86f-e0ff-400e-93ca-84a26a630e16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:07.160 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-18T01:16:07.723 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:16:07.722+0000 7fac23618640 -1 Falling back to public interface 2024-09-18T01:16:07.977 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:07.977+0000 7feae7d2d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T01:16:08.033 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:08.032+0000 7feae7d2d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T01:16:08.183 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:16:08.183+0000 7fa23ded4640 -1 Falling back to public interface 2024-09-18T01:16:08.207 INFO:tasks.ceph.cluster1.osd.1.smithi066.stderr:2024-09-18T01:16:08.206+0000 7f58565b2640 -1 Falling back to public interface 2024-09-18T01:16:08.421 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:08.421+0000 7feae7d2d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-18T01:16:08.511 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:08.510+0000 7feae7d2d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T01:16:08.623 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:08.622+0000 7feae7d2d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T01:16:08.671 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:08.670+0000 7feae7d2d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T01:16:08.789 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:08.789+0000 7feae7d2d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-18T01:16:08.845 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:08.844+0000 7feae7d2d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T01:16:08.892 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:08.891+0000 7feae7d2d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T01:16:09.010 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:09.009+0000 7feae7d2d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T01:16:09.062 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:09.061+0000 7feae7d2d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T01:16:09.116 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:09.115+0000 7feae7d2d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T01:16:09.186 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:09.186+0000 7feae7d2d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T01:16:09.233 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:09.232+0000 7feae7d2d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T01:16:09.279 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:09.279+0000 7feae7d2d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T01:16:09.964 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:09.964+0000 7feae7d2d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T01:16:10.972 INFO:tasks.ceph.cluster1.mgr.x.smithi066.stderr:2024-09-18T01:16:10.971+0000 7fea885f5640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-18T01:16:11.755 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:16:11.755+0000 7fac23618640 -1 osd.2 0 log_to_monitors true 2024-09-18T01:16:11.987 INFO:tasks.ceph.cluster1.osd.1.smithi066.stderr:2024-09-18T01:16:11.986+0000 7f58565b2640 -1 osd.1 0 log_to_monitors true 2024-09-18T01:16:12.449 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:16:12.446+0000 7fa23ded4640 -1 osd.0 0 log_to_monitors true 2024-09-18T01:16:13.464 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-18T01:16:13.726 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:16:13.726+0000 7fac1ef7a640 -1 osd.2 0 waiting for initial osdmap 2024-09-18T01:16:13.735 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:16:13.735+0000 7fac16d9a640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T01:16:13.815 INFO:teuthology.misc.health.smithi066.stdout: 2024-09-18T01:16:13.815 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":7,"fsid":"5799928d-af76-40cf-a91d-c6362506fa2c","created":"2024-09-18T01:16:05.571663+0000","modified":"2024-09-18T01:16:12.970902+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T01:16:06.473310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"81a7cd46-8949-4110-bb1e-add42dfb2660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88aec54c-891e-4b4f-967d-f7bb2e7d4c71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2630c86f-e0ff-400e-93ca-84a26a630e16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:13.826 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-18T01:16:14.917 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:16:14.916+0000 7fa239e63640 -1 osd.0 0 waiting for initial osdmap 2024-09-18T01:16:14.923 INFO:tasks.ceph.cluster1.osd.1.smithi066.stderr:2024-09-18T01:16:14.922+0000 7f5852d55640 -1 osd.1 0 waiting for initial osdmap 2024-09-18T01:16:14.930 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:16:14.930+0000 7fa231c83640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T01:16:14.935 INFO:tasks.ceph.cluster1.osd.1.smithi066.stderr:2024-09-18T01:16:14.935+0000 7f584a361640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T01:16:20.129 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-18T01:16:20.423 INFO:teuthology.misc.health.smithi066.stdout: 2024-09-18T01:16:20.423 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":12,"fsid":"5799928d-af76-40cf-a91d-c6362506fa2c","created":"2024-09-18T01:16:05.571663+0000","modified":"2024-09-18T01:16:18.020242+0000","last_up_change":"2024-09-18T01:16:14.978708+0000","last_in_change":"2024-09-18T01:16:06.473310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T01:16:16.014269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"81a7cd46-8949-4110-bb1e-add42dfb2660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6809","nonce":3770637869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6811","nonce":3770637869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6815","nonce":3770637869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6813","nonce":3770637869}]},"public_addr":"172.21.15.66:6809/3770637869","cluster_addr":"172.21.15.66:6811/3770637869","heartbeat_back_addr":"172.21.15.66:6815/3770637869","heartbeat_front_addr":"172.21.15.66:6813/3770637869","state":["exists","up"]},{"osd":1,"uuid":"88aec54c-891e-4b4f-967d-f7bb2e7d4c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6817","nonce":2946278758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6819","nonce":2946278758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6823","nonce":2946278758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6821","nonce":2946278758}]},"public_addr":"172.21.15.66:6817/2946278758","cluster_addr":"172.21.15.66:6819/2946278758","heartbeat_back_addr":"172.21.15.66:6823/2946278758","heartbeat_front_addr":"172.21.15.66:6821/2946278758","state":["exists","up"]},{"osd":2,"uuid":"2630c86f-e0ff-400e-93ca-84a26a630e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6801","nonce":1507114421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6803","nonce":1507114421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6807","nonce":1507114421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6805","nonce":1507114421}]},"public_addr":"172.21.15.66:6801/1507114421","cluster_addr":"172.21.15.66:6803/1507114421","heartbeat_back_addr":"172.21.15.66:6807/1507114421","heartbeat_front_addr":"172.21.15.66:6805/1507114421","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:13.460411+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:13.004326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:12.762131+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:20.434 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-18T01:16:20.434 INFO:tasks.ceph:Creating RBD pool 2024-09-18T01:16:20.434 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-18T01:16:21.029 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'rbd' created 2024-09-18T01:16:21.045 DEBUG:teuthology.orchestra.run.smithi066:> rbd --cluster cluster1 pool init rbd 2024-09-18T01:16:24.054 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-18T01:16:24.055 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-18T01:16:24.055 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-18T01:16:24.055 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-18T01:16:24.056 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T01:16:24.404 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:24.416 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3404922931},{"type":"v1","addr":"172.21.15.66:6825","nonce":3404922931}]},"active_addr":"172.21.15.66:6825/3404922931","active_change":"2024-09-18T01:16:09.965488+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":143542539}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":1300902156}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":4108868302}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":3707955135}]}]} 2024-09-18T01:16:24.418 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-18T01:16:24.418 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-18T01:16:24.418 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-18T01:16:24.709 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:24.709 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":16,"fsid":"5799928d-af76-40cf-a91d-c6362506fa2c","created":"2024-09-18T01:16:05.571663+0000","modified":"2024-09-18T01:16:24.041345+0000","last_up_change":"2024-09-18T01:16:14.978708+0000","last_in_change":"2024-09-18T01:16:06.473310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T01:16:16.014269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T01:16:20.752157+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"81a7cd46-8949-4110-bb1e-add42dfb2660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6809","nonce":3770637869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6811","nonce":3770637869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6815","nonce":3770637869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6813","nonce":3770637869}]},"public_addr":"172.21.15.66:6809/3770637869","cluster_addr":"172.21.15.66:6811/3770637869","heartbeat_back_addr":"172.21.15.66:6815/3770637869","heartbeat_front_addr":"172.21.15.66:6813/3770637869","state":["exists","up"]},{"osd":1,"uuid":"88aec54c-891e-4b4f-967d-f7bb2e7d4c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6817","nonce":2946278758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6819","nonce":2946278758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6823","nonce":2946278758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6821","nonce":2946278758}]},"public_addr":"172.21.15.66:6817/2946278758","cluster_addr":"172.21.15.66:6819/2946278758","heartbeat_back_addr":"172.21.15.66:6823/2946278758","heartbeat_front_addr":"172.21.15.66:6821/2946278758","state":["exists","up"]},{"osd":2,"uuid":"2630c86f-e0ff-400e-93ca-84a26a630e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6801","nonce":1507114421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6803","nonce":1507114421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6807","nonce":1507114421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6805","nonce":1507114421}]},"public_addr":"172.21.15.66:6801/1507114421","cluster_addr":"172.21.15.66:6803/1507114421","heartbeat_back_addr":"172.21.15.66:6807/1507114421","heartbeat_front_addr":"172.21.15.66:6805/1507114421","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:13.460411+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:13.004326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:12.762131+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:24.721 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-18T01:16:24.722 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-18T01:16:25.011 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:25.011 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":16,"fsid":"5799928d-af76-40cf-a91d-c6362506fa2c","created":"2024-09-18T01:16:05.571663+0000","modified":"2024-09-18T01:16:24.041345+0000","last_up_change":"2024-09-18T01:16:14.978708+0000","last_in_change":"2024-09-18T01:16:06.473310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T01:16:16.014269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T01:16:20.752157+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"81a7cd46-8949-4110-bb1e-add42dfb2660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6809","nonce":3770637869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6811","nonce":3770637869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6815","nonce":3770637869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6813","nonce":3770637869}]},"public_addr":"172.21.15.66:6809/3770637869","cluster_addr":"172.21.15.66:6811/3770637869","heartbeat_back_addr":"172.21.15.66:6815/3770637869","heartbeat_front_addr":"172.21.15.66:6813/3770637869","state":["exists","up"]},{"osd":1,"uuid":"88aec54c-891e-4b4f-967d-f7bb2e7d4c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6817","nonce":2946278758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6819","nonce":2946278758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6823","nonce":2946278758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6821","nonce":2946278758}]},"public_addr":"172.21.15.66:6817/2946278758","cluster_addr":"172.21.15.66:6819/2946278758","heartbeat_back_addr":"172.21.15.66:6823/2946278758","heartbeat_front_addr":"172.21.15.66:6821/2946278758","state":["exists","up"]},{"osd":2,"uuid":"2630c86f-e0ff-400e-93ca-84a26a630e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6801","nonce":1507114421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6803","nonce":1507114421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6807","nonce":1507114421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6805","nonce":1507114421}]},"public_addr":"172.21.15.66:6801/1507114421","cluster_addr":"172.21.15.66:6803/1507114421","heartbeat_back_addr":"172.21.15.66:6807/1507114421","heartbeat_front_addr":"172.21.15.66:6805/1507114421","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:13.460411+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:13.004326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:12.762131+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:25.024 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-18T01:16:25.025 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-18T01:16:25.025 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-18T01:16:25.168 INFO:teuthology.orchestra.run.smithi066.stdout:38654705668 2024-09-18T01:16:25.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-18T01:16:25.172 INFO:teuthology.orchestra.run.smithi066.stdout:38654705668 2024-09-18T01:16:25.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-18T01:16:25.173 INFO:teuthology.orchestra.run.smithi066.stdout:34359738374 2024-09-18T01:16:25.173 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-18T01:16:25.480 INFO:teuthology.orchestra.run.smithi066.stdout:38654705667 2024-09-18T01:16:25.484 INFO:teuthology.orchestra.run.smithi066.stdout:34359738372 2024-09-18T01:16:25.490 INFO:teuthology.orchestra.run.smithi066.stdout:38654705667 2024-09-18T01:16:25.493 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705667 for osd.1 2024-09-18T01:16:25.499 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.2 2024-09-18T01:16:25.504 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705667 for osd.0 2024-09-18T01:16:26.495 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-18T01:16:26.500 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-18T01:16:26.504 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-18T01:16:26.829 INFO:teuthology.orchestra.run.smithi066.stdout:38654705668 2024-09-18T01:16:26.846 INFO:teuthology.orchestra.run.smithi066.stdout:34359738374 2024-09-18T01:16:26.847 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705668 for osd.1 2024-09-18T01:16:26.847 DEBUG:teuthology.parallel:result is None 2024-09-18T01:16:26.858 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2024-09-18T01:16:26.858 DEBUG:teuthology.parallel:result is None 2024-09-18T01:16:26.869 INFO:teuthology.orchestra.run.smithi066.stdout:38654705668 2024-09-18T01:16:26.882 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705668 for osd.0 2024-09-18T01:16:26.882 DEBUG:teuthology.parallel:result is None 2024-09-18T01:16:26.882 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-18T01:16:26.882 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T01:16:27.211 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:27.212 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-09-18T01:16:27.223 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-18T01:16:25.971515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81632,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033888,"statfs":{"total":289910292480,"available":289826701312,"internally_reserved":0,"allocated":548864,"data_stored":1302991,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939128"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.048908+0000","last_change":"2024-09-18T01:16:24.048908+0000","last_active":"2024-09-18T01:16:24.048908+0000","last_peered":"2024-09-18T01:16:24.048908+0000","last_clean":"2024-09-18T01:16:24.048908+0000","last_became_active":"2024-09-18T01:16:22.049384+0000","last_became_peered":"2024-09-18T01:16:22.049384+0000","last_unstale":"2024-09-18T01:16:24.048908+0000","last_undegraded":"2024-09-18T01:16:24.048908+0000","last_fullsized":"2024-09-18T01:16:24.048908+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:08:44.387269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067731499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.049185+0000","last_change":"2024-09-18T01:16:24.049185+0000","last_active":"2024-09-18T01:16:24.049185+0000","last_peered":"2024-09-18T01:16:24.049185+0000","last_clean":"2024-09-18T01:16:24.049185+0000","last_became_active":"2024-09-18T01:16:22.048383+0000","last_became_peered":"2024-09-18T01:16:22.048383+0000","last_unstale":"2024-09-18T01:16:24.049185+0000","last_undegraded":"2024-09-18T01:16:24.049185+0000","last_fullsized":"2024-09-18T01:16:24.049185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:33:41.916036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076761799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.048791+0000","last_change":"2024-09-18T01:16:24.048791+0000","last_active":"2024-09-18T01:16:24.048791+0000","last_peered":"2024-09-18T01:16:24.048791+0000","last_clean":"2024-09-18T01:16:24.048791+0000","last_became_active":"2024-09-18T01:16:22.049251+0000","last_became_peered":"2024-09-18T01:16:22.049251+0000","last_unstale":"2024-09-18T01:16:24.048791+0000","last_undegraded":"2024-09-18T01:16:24.048791+0000","last_fullsized":"2024-09-18T01:16:24.048791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:25.091421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057751099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.049406+0000","last_change":"2024-09-18T01:16:24.049406+0000","last_active":"2024-09-18T01:16:24.049406+0000","last_peered":"2024-09-18T01:16:24.049406+0000","last_clean":"2024-09-18T01:16:24.049406+0000","last_became_active":"2024-09-18T01:16:22.049287+0000","last_became_peered":"2024-09-18T01:16:22.049287+0000","last_unstale":"2024-09-18T01:16:24.049406+0000","last_undegraded":"2024-09-18T01:16:24.049406+0000","last_fullsized":"2024-09-18T01:16:24.049406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:51:35.915729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001186821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.047739+0000","last_change":"2024-09-18T01:16:24.045936+0000","last_active":"2024-09-18T01:16:24.047739+0000","last_peered":"2024-09-18T01:16:24.047739+0000","last_clean":"2024-09-18T01:16:24.047739+0000","last_became_active":"2024-09-18T01:16:22.048463+0000","last_became_peered":"2024-09-18T01:16:22.048463+0000","last_unstale":"2024-09-18T01:16:24.047739+0000","last_undegraded":"2024-09-18T01:16:24.047739+0000","last_fullsized":"2024-09-18T01:16:24.047739+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:09:27.308691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050943500000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.060226+0000","last_change":"2024-09-18T01:16:24.060226+0000","last_active":"2024-09-18T01:16:24.060226+0000","last_peered":"2024-09-18T01:16:24.060226+0000","last_clean":"2024-09-18T01:16:24.060226+0000","last_became_active":"2024-09-18T01:16:22.048849+0000","last_became_peered":"2024-09-18T01:16:22.048849+0000","last_unstale":"2024-09-18T01:16:24.060226+0000","last_undegraded":"2024-09-18T01:16:24.060226+0000","last_fullsized":"2024-09-18T01:16:24.060226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:32.872881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056915500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.060127+0000","last_change":"2024-09-18T01:16:24.060127+0000","last_active":"2024-09-18T01:16:24.060127+0000","last_peered":"2024-09-18T01:16:24.060127+0000","last_clean":"2024-09-18T01:16:24.060127+0000","last_became_active":"2024-09-18T01:16:22.045793+0000","last_became_peered":"2024-09-18T01:16:22.045793+0000","last_unstale":"2024-09-18T01:16:24.060127+0000","last_undegraded":"2024-09-18T01:16:24.060127+0000","last_fullsized":"2024-09-18T01:16:24.060127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:08:15.672511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048395300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.049520+0000","last_change":"2024-09-18T01:16:24.049520+0000","last_active":"2024-09-18T01:16:24.049520+0000","last_peered":"2024-09-18T01:16:24.049520+0000","last_clean":"2024-09-18T01:16:24.049520+0000","last_became_active":"2024-09-18T01:16:22.045810+0000","last_became_peered":"2024-09-18T01:16:22.045810+0000","last_unstale":"2024-09-18T01:16:24.049520+0000","last_undegraded":"2024-09-18T01:16:24.049520+0000","last_fullsized":"2024-09-18T01:16:24.049520+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:33.391698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010551429999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.048063+0000","last_change":"2024-09-18T01:16:17.153205+0000","last_active":"2024-09-18T01:16:24.048063+0000","last_peered":"2024-09-18T01:16:24.048063+0000","last_clean":"2024-09-18T01:16:24.048063+0000","last_became_active":"2024-09-18T01:16:17.151939+0000","last_became_peered":"2024-09-18T01:16:17.151939+0000","last_unstale":"2024-09-18T01:16:24.048063+0000","last_undegraded":"2024-09-18T01:16:24.048063+0000","last_fullsized":"2024-09-18T01:16:24.048063+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_clean_scrub_stamp":"2024-09-18T01:16:16.017299+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:51:08.601096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":631126,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":204800,"data_stored":631126,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":139264,"data_stored":40739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:16:27.224 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T01:16:27.510 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:27.511 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-09-18T01:16:27.522 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-18T01:16:25.971515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81632,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033888,"statfs":{"total":289910292480,"available":289826701312,"internally_reserved":0,"allocated":548864,"data_stored":1302991,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939128"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.048908+0000","last_change":"2024-09-18T01:16:24.048908+0000","last_active":"2024-09-18T01:16:24.048908+0000","last_peered":"2024-09-18T01:16:24.048908+0000","last_clean":"2024-09-18T01:16:24.048908+0000","last_became_active":"2024-09-18T01:16:22.049384+0000","last_became_peered":"2024-09-18T01:16:22.049384+0000","last_unstale":"2024-09-18T01:16:24.048908+0000","last_undegraded":"2024-09-18T01:16:24.048908+0000","last_fullsized":"2024-09-18T01:16:24.048908+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:08:44.387269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067731499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.049185+0000","last_change":"2024-09-18T01:16:24.049185+0000","last_active":"2024-09-18T01:16:24.049185+0000","last_peered":"2024-09-18T01:16:24.049185+0000","last_clean":"2024-09-18T01:16:24.049185+0000","last_became_active":"2024-09-18T01:16:22.048383+0000","last_became_peered":"2024-09-18T01:16:22.048383+0000","last_unstale":"2024-09-18T01:16:24.049185+0000","last_undegraded":"2024-09-18T01:16:24.049185+0000","last_fullsized":"2024-09-18T01:16:24.049185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:33:41.916036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076761799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.048791+0000","last_change":"2024-09-18T01:16:24.048791+0000","last_active":"2024-09-18T01:16:24.048791+0000","last_peered":"2024-09-18T01:16:24.048791+0000","last_clean":"2024-09-18T01:16:24.048791+0000","last_became_active":"2024-09-18T01:16:22.049251+0000","last_became_peered":"2024-09-18T01:16:22.049251+0000","last_unstale":"2024-09-18T01:16:24.048791+0000","last_undegraded":"2024-09-18T01:16:24.048791+0000","last_fullsized":"2024-09-18T01:16:24.048791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:25.091421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057751099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.049406+0000","last_change":"2024-09-18T01:16:24.049406+0000","last_active":"2024-09-18T01:16:24.049406+0000","last_peered":"2024-09-18T01:16:24.049406+0000","last_clean":"2024-09-18T01:16:24.049406+0000","last_became_active":"2024-09-18T01:16:22.049287+0000","last_became_peered":"2024-09-18T01:16:22.049287+0000","last_unstale":"2024-09-18T01:16:24.049406+0000","last_undegraded":"2024-09-18T01:16:24.049406+0000","last_fullsized":"2024-09-18T01:16:24.049406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:51:35.915729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001186821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.047739+0000","last_change":"2024-09-18T01:16:24.045936+0000","last_active":"2024-09-18T01:16:24.047739+0000","last_peered":"2024-09-18T01:16:24.047739+0000","last_clean":"2024-09-18T01:16:24.047739+0000","last_became_active":"2024-09-18T01:16:22.048463+0000","last_became_peered":"2024-09-18T01:16:22.048463+0000","last_unstale":"2024-09-18T01:16:24.047739+0000","last_undegraded":"2024-09-18T01:16:24.047739+0000","last_fullsized":"2024-09-18T01:16:24.047739+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:09:27.308691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050943500000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.060226+0000","last_change":"2024-09-18T01:16:24.060226+0000","last_active":"2024-09-18T01:16:24.060226+0000","last_peered":"2024-09-18T01:16:24.060226+0000","last_clean":"2024-09-18T01:16:24.060226+0000","last_became_active":"2024-09-18T01:16:22.048849+0000","last_became_peered":"2024-09-18T01:16:22.048849+0000","last_unstale":"2024-09-18T01:16:24.060226+0000","last_undegraded":"2024-09-18T01:16:24.060226+0000","last_fullsized":"2024-09-18T01:16:24.060226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:32.872881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056915500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.060127+0000","last_change":"2024-09-18T01:16:24.060127+0000","last_active":"2024-09-18T01:16:24.060127+0000","last_peered":"2024-09-18T01:16:24.060127+0000","last_clean":"2024-09-18T01:16:24.060127+0000","last_became_active":"2024-09-18T01:16:22.045793+0000","last_became_peered":"2024-09-18T01:16:22.045793+0000","last_unstale":"2024-09-18T01:16:24.060127+0000","last_undegraded":"2024-09-18T01:16:24.060127+0000","last_fullsized":"2024-09-18T01:16:24.060127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:08:15.672511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048395300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.049520+0000","last_change":"2024-09-18T01:16:24.049520+0000","last_active":"2024-09-18T01:16:24.049520+0000","last_peered":"2024-09-18T01:16:24.049520+0000","last_clean":"2024-09-18T01:16:24.049520+0000","last_became_active":"2024-09-18T01:16:22.045810+0000","last_became_peered":"2024-09-18T01:16:22.045810+0000","last_unstale":"2024-09-18T01:16:24.049520+0000","last_undegraded":"2024-09-18T01:16:24.049520+0000","last_fullsized":"2024-09-18T01:16:24.049520+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:33.391698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010551429999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:16:24.048063+0000","last_change":"2024-09-18T01:16:17.153205+0000","last_active":"2024-09-18T01:16:24.048063+0000","last_peered":"2024-09-18T01:16:24.048063+0000","last_clean":"2024-09-18T01:16:24.048063+0000","last_became_active":"2024-09-18T01:16:17.151939+0000","last_became_peered":"2024-09-18T01:16:17.151939+0000","last_unstale":"2024-09-18T01:16:24.048063+0000","last_undegraded":"2024-09-18T01:16:24.048063+0000","last_fullsized":"2024-09-18T01:16:24.048063+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_clean_scrub_stamp":"2024-09-18T01:16:16.017299+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:51:08.601096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":631126,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":204800,"data_stored":631126,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":139264,"data_stored":40739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:16:27.523 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-18T01:16:27.523 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-18T01:16:27.524 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-18T01:16:27.524 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-18T01:16:27.846 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:27.846 INFO:teuthology.orchestra.run.smithi066.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T01:16:27.858 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-18T01:16:27.858 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T01:16:27.867 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-18T01:16:27.867 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T01:16:27.868 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909706', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1983, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'sleep_before_teardown': 0, 'subset': '79/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7', 'targets': {'smithi066.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGn6aYBEPHizdpZJ+69qOLn/mMp4MZTTPpuLz8kHDglD/5eyJWhiY+WovdoH4DdROl74VGgdBmBtT4aCUx/yNCU=', 'smithi115.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbKvIrtgNKXW7OrMynfsNnBa2DwHxfRxcT/MU/kRBzd2q9W/ve+QPw3O5ahdvSfbu3T2PgayQrYyvuW1+ta3Fc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T01:16:27.868 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-18T01:16:27.871 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-18T01:16:27.890 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T01:16:27.927 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T01:16:27.958 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:27.958 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T01:16:28.016 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T01:16:28.016 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-09-18 01:15:44.107996930 +0000 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-09-18 01:13:01.334384096 +0000 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-09-18 01:13:01.334384096 +0000 2024-09-18T01:16:28.074 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-09-18 01:13:01.334384096 +0000 2024-09-18T01:16:28.074 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T01:16:28.141 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-09-18T01:16:28.141 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-09-18T01:16:28.141 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000247973 s, 2.1 MB/s 2024-09-18T01:16:28.142 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T01:16:28.202 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-09-18 01:15:44.107996930 +0000 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-09-18 01:13:01.659377976 +0000 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-09-18 01:13:01.659377976 +0000 2024-09-18T01:16:28.259 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-09-18 01:13:01.659377976 +0000 2024-09-18T01:16:28.260 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T01:16:28.324 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-09-18T01:16:28.324 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-09-18T01:16:28.324 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000221643 s, 2.3 MB/s 2024-09-18T01:16:28.325 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T01:16:28.382 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2024-09-18T01:16:28.437 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T01:16:28.437 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T01:16:28.437 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-18T01:16:28.437 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T01:16:28.437 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T01:16:28.437 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-09-18 01:15:44.107996930 +0000 2024-09-18T01:16:28.437 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-09-18 01:13:01.981371913 +0000 2024-09-18T01:16:28.438 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-09-18 01:13:01.981371913 +0000 2024-09-18T01:16:28.438 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-09-18 01:13:01.981371913 +0000 2024-09-18T01:16:28.438 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T01:16:28.501 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-09-18T01:16:28.502 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-09-18T01:16:28.502 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000165347 s, 3.1 MB/s 2024-09-18T01:16:28.503 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T01:16:28.559 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2024-09-18T01:16:28.614 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T01:16:28.614 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T01:16:28.614 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-18T01:16:28.614 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T01:16:28.614 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T01:16:28.614 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-09-18 01:15:44.108996911 +0000 2024-09-18T01:16:28.614 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-09-18 01:13:02.309365736 +0000 2024-09-18T01:16:28.615 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-09-18 01:13:02.309365736 +0000 2024-09-18T01:16:28.615 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-09-18 01:13:02.309365736 +0000 2024-09-18T01:16:28.615 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T01:16:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-09-18T01:16:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-09-18T01:16:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000202688 s, 2.5 MB/s 2024-09-18T01:16:28.680 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T01:16:28.737 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T01:16:28.737 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-18T01:16:28.737 INFO:tasks.ceph:Generating config... 2024-09-18T01:16:28.738 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-18T01:16:28.738 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-18T01:16:28.738 INFO:tasks.ceph:[client] log to stderr = False 2024-09-18T01:16:28.738 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-18T01:16:28.738 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-18T01:16:28.738 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:28.738 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:28.738 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:28.739 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:28.739 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:28.739 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:28.739 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:28.739 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:28.739 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T01:16:28.739 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-18T01:16:28.739 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-18T01:16:28.739 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T01:16:28.739 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T01:16:28.739 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T01:16:28.739 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T01:16:28.739 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T01:16:28.739 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T01:16:28.739 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T01:16:28.740 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T01:16:28.740 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-18T01:16:28.741 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-18T01:16:28.817 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.keyring 2024-09-18T01:16:28.819 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-18T01:16:28.903 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-18T01:16:28.968 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.115')] 2024-09-18T01:16:28.968 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.115', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-18T01:16:28.969 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:28.969 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T01:16:29.023 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.115 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-18T01:16:29.103 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-18T01:16:29.103 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: generated fsid fcf9d702-6ece-4878-bfd9-543b84db5aa5 2024-09-18T01:16:29.103 INFO:teuthology.orchestra.run.smithi115.stdout:setting min_mon_release = quincy 2024-09-18T01:16:29.104 INFO:teuthology.orchestra.run.smithi115.stdout:epoch 0 2024-09-18T01:16:29.104 INFO:teuthology.orchestra.run.smithi115.stdout:fsid fcf9d702-6ece-4878-bfd9-543b84db5aa5 2024-09-18T01:16:29.104 INFO:teuthology.orchestra.run.smithi115.stdout:last_changed 2024-09-18T01:16:29.103646+0000 2024-09-18T01:16:29.104 INFO:teuthology.orchestra.run.smithi115.stdout:created 2024-09-18T01:16:29.103646+0000 2024-09-18T01:16:29.104 INFO:teuthology.orchestra.run.smithi115.stdout:min_mon_release 17 (quincy) 2024-09-18T01:16:29.104 INFO:teuthology.orchestra.run.smithi115.stdout:election_strategy: 1 2024-09-18T01:16:29.104 INFO:teuthology.orchestra.run.smithi115.stdout:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-18T01:16:29.104 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-18T01:16:29.105 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T01:16:29.161 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID fcf9d702-6ece-4878-bfd9-543b84db5aa5... 2024-09-18T01:16:29.162 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-18T01:16:29.165 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout:[global] 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: chdir = "" 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: auth supported = cephx 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: filestore xattr use omap = true 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: mon clock drift allowed = 1.000 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: osd crush chooseleaf type = 0 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: auth debug = true 2024-09-18T01:16:29.216 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on old message = true 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on bug = true 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: mon pg warn max object skew = 0 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default size = 2 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.217 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow primary affinity = true 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow pg remap = true 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on legacy crush tunables = false 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on crush straw calc version zero = false 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on no sortbitwise = false 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on osd down out interval zero = false 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on too few osds = false 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon_allow_pool_size_one = true 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: osd default data pool replay window = 5 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.218 INFO:teuthology.orchestra.run.smithi066.stdout: mon allow pool delete = true 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: mon cluster log file level = debug 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: debug asserts on shutdown = true 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: mon health detail to clog = false 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: mon host = 172.21.15.115 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: mon client directed command retry = 5 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on pool no app = False 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: ms inject socket failures = 5000 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: fsid = fcf9d702-6ece-4878-bfd9-543b84db5aa5 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout:[osd] 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: osd journal size = 100 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub load threshold = 5.0 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub max interval = 600 2024-09-18T01:16:29.219 INFO:teuthology.orchestra.run.smithi066.stdout: osd mclock profile = high_recovery_ops 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: osd recover clone overlap = true 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: osd recovery max chunk = 1048576 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug shutdown = true 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug op order = true 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug verify stray on activate = true 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug trim objects = true 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.220 INFO:teuthology.orchestra.run.smithi066.stdout: osd open classes on start = true 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug pg log writeout = true 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: osd deep scrub update digest min age = 30 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: osd map max advance = 10 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: journal zero on create = true 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: filestore ondisk finisher threads = 3 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: filestore apply finisher threads = 3 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: bdev debug aio = true 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug misdirected ops = true 2024-09-18T01:16:29.221 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore block size = 96636764160 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore compression algorithm = snappy 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore compression mode = aggressive 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore fsck on mount = True 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: debug bluefs = 1/20 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: debug bluestore = 1/20 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: debug osd = 20 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: debug rocksdb = 4/10 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd full ratio = 0.9 2024-09-18T01:16:29.222 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd nearfull ratio = 0.8 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: osd failsafe full ratio = 0.95 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: osd objectstore = bluestore 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr] 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: debug mgr = 20 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: mgr/telemetry/nag = false 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout:[mon] 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2024-09-18T01:16:29.223 INFO:teuthology.orchestra.run.smithi066.stdout: debug paxos = 20 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: mon data avail warn = 5 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: mon mgr mkfs grace = 240 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd reporter subtree level = osd 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd prime pg temp = true 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T01:16:29.224 INFO:teuthology.orchestra.run.smithi066.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T01:16:29.225 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.225 INFO:teuthology.orchestra.run.smithi066.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T01:16:29.225 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T01:16:29.225 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T01:16:29.225 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.225 INFO:teuthology.orchestra.run.smithi066.stdout: # 1m isn't quite enough 2024-09-18T01:16:29.225 INFO:teuthology.orchestra.run.smithi066.stdout: mon_down_mkfs_grace = 2m 2024-09-18T01:16:29.225 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_filestore_osds = false 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout:[client] 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: rgw cache enabled = true 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable ops log = true 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable usage log = true 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: debug rbd = 20 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: debug rbd_mirror = 15 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: log to stderr = False 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: rbd default features = 125 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: rbd mirror image policy type = none 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.a] 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.0] 2024-09-18T01:16:29.226 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.1] 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.2] 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout:[client.mirror.3] 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:29.227 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:29.245 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2024-09-18T01:16:29.245 INFO:teuthology.orchestra.run.smithi115.stdout: chdir = "" 2024-09-18T01:16:29.245 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T01:16:29.245 INFO:teuthology.orchestra.run.smithi115.stdout: auth supported = cephx 2024-09-18T01:16:29.245 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.245 INFO:teuthology.orchestra.run.smithi115.stdout: filestore xattr use omap = true 2024-09-18T01:16:29.245 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: mon clock drift allowed = 1.000 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: osd crush chooseleaf type = 0 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: auth debug = true 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on old message = true 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on bug = true 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: mon pg warn max object skew = 0 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T01:16:29.246 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default size = 2 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow primary affinity = true 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow pg remap = true 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on legacy crush tunables = false 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on crush straw calc version zero = false 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on no sortbitwise = false 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on osd down out interval zero = false 2024-09-18T01:16:29.247 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on too few osds = false 2024-09-18T01:16:29.248 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T01:16:29.248 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T01:16:29.248 INFO:teuthology.orchestra.run.smithi115.stdout: mon_allow_pool_size_one = true 2024-09-18T01:16:29.248 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.248 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T01:16:29.248 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.249 INFO:teuthology.orchestra.run.smithi115.stdout: osd default data pool replay window = 5 2024-09-18T01:16:29.249 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.249 INFO:teuthology.orchestra.run.smithi115.stdout: mon allow pool delete = true 2024-09-18T01:16:29.249 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.249 INFO:teuthology.orchestra.run.smithi115.stdout: mon cluster log file level = debug 2024-09-18T01:16:29.249 INFO:teuthology.orchestra.run.smithi115.stdout: debug asserts on shutdown = true 2024-09-18T01:16:29.249 INFO:teuthology.orchestra.run.smithi115.stdout: mon health detail to clog = false 2024-09-18T01:16:29.250 INFO:teuthology.orchestra.run.smithi115.stdout: mon host = 172.21.15.115 2024-09-18T01:16:29.250 INFO:teuthology.orchestra.run.smithi115.stdout: mon client directed command retry = 5 2024-09-18T01:16:29.250 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on pool no app = False 2024-09-18T01:16:29.250 INFO:teuthology.orchestra.run.smithi115.stdout: ms inject socket failures = 5000 2024-09-18T01:16:29.250 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = fcf9d702-6ece-4878-bfd9-543b84db5aa5 2024-09-18T01:16:29.250 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.251 INFO:teuthology.orchestra.run.smithi115.stdout:[osd] 2024-09-18T01:16:29.251 INFO:teuthology.orchestra.run.smithi115.stdout: osd journal size = 100 2024-09-18T01:16:29.251 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.251 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub load threshold = 5.0 2024-09-18T01:16:29.251 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub max interval = 600 2024-09-18T01:16:29.251 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock profile = high_recovery_ops 2024-09-18T01:16:29.252 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.252 INFO:teuthology.orchestra.run.smithi115.stdout: osd recover clone overlap = true 2024-09-18T01:16:29.252 INFO:teuthology.orchestra.run.smithi115.stdout: osd recovery max chunk = 1048576 2024-09-18T01:16:29.252 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.252 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug shutdown = true 2024-09-18T01:16:29.252 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug op order = true 2024-09-18T01:16:29.253 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify stray on activate = true 2024-09-18T01:16:29.253 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug trim objects = true 2024-09-18T01:16:29.253 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.253 INFO:teuthology.orchestra.run.smithi115.stdout: osd open classes on start = true 2024-09-18T01:16:29.253 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug pg log writeout = true 2024-09-18T01:16:29.253 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.254 INFO:teuthology.orchestra.run.smithi115.stdout: osd deep scrub update digest min age = 30 2024-09-18T01:16:29.254 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.254 INFO:teuthology.orchestra.run.smithi115.stdout: osd map max advance = 10 2024-09-18T01:16:29.254 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.254 INFO:teuthology.orchestra.run.smithi115.stdout: journal zero on create = true 2024-09-18T01:16:29.254 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.255 INFO:teuthology.orchestra.run.smithi115.stdout: filestore ondisk finisher threads = 3 2024-09-18T01:16:29.255 INFO:teuthology.orchestra.run.smithi115.stdout: filestore apply finisher threads = 3 2024-09-18T01:16:29.255 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.255 INFO:teuthology.orchestra.run.smithi115.stdout: bdev debug aio = true 2024-09-18T01:16:29.255 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug misdirected ops = true 2024-09-18T01:16:29.255 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore block size = 96636764160 2024-09-18T01:16:29.256 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore compression algorithm = snappy 2024-09-18T01:16:29.256 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore compression mode = aggressive 2024-09-18T01:16:29.256 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore fsck on mount = True 2024-09-18T01:16:29.256 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluefs = 1/20 2024-09-18T01:16:29.256 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluestore = 1/20 2024-09-18T01:16:29.256 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T01:16:29.257 INFO:teuthology.orchestra.run.smithi115.stdout: debug osd = 20 2024-09-18T01:16:29.257 INFO:teuthology.orchestra.run.smithi115.stdout: debug rocksdb = 4/10 2024-09-18T01:16:29.257 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T01:16:29.257 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd full ratio = 0.9 2024-09-18T01:16:29.257 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd nearfull ratio = 0.8 2024-09-18T01:16:29.257 INFO:teuthology.orchestra.run.smithi115.stdout: osd failsafe full ratio = 0.95 2024-09-18T01:16:29.258 INFO:teuthology.orchestra.run.smithi115.stdout: osd objectstore = bluestore 2024-09-18T01:16:29.258 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.258 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr] 2024-09-18T01:16:29.258 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T01:16:29.258 INFO:teuthology.orchestra.run.smithi115.stdout: debug mgr = 20 2024-09-18T01:16:29.260 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2024-09-18T01:16:29.260 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2024-09-18T01:16:29.260 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2024-09-18T01:16:29.260 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2024-09-18T01:16:29.260 INFO:teuthology.orchestra.run.smithi115.stdout: mgr/telemetry/nag = false 2024-09-18T01:16:29.261 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.261 INFO:teuthology.orchestra.run.smithi115.stdout:[mon] 2024-09-18T01:16:29.261 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T01:16:29.261 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2024-09-18T01:16:29.261 INFO:teuthology.orchestra.run.smithi115.stdout: debug paxos = 20 2024-09-18T01:16:29.261 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2024-09-18T01:16:29.262 INFO:teuthology.orchestra.run.smithi115.stdout: mon data avail warn = 5 2024-09-18T01:16:29.262 INFO:teuthology.orchestra.run.smithi115.stdout: mon mgr mkfs grace = 240 2024-09-18T01:16:29.262 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2024-09-18T01:16:29.262 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd reporter subtree level = osd 2024-09-18T01:16:29.262 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd prime pg temp = true 2024-09-18T01:16:29.262 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2024-09-18T01:16:29.262 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.263 INFO:teuthology.orchestra.run.smithi115.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T01:16:29.263 INFO:teuthology.orchestra.run.smithi115.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T01:16:29.263 INFO:teuthology.orchestra.run.smithi115.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T01:16:29.263 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.263 INFO:teuthology.orchestra.run.smithi115.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T01:16:29.263 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T01:16:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T01:16:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: # 1m isn't quite enough 2024-09-18T01:16:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: mon_down_mkfs_grace = 2m 2024-09-18T01:16:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_filestore_osds = false 2024-09-18T01:16:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:29.265 INFO:teuthology.orchestra.run.smithi115.stdout:[client] 2024-09-18T01:16:29.265 INFO:teuthology.orchestra.run.smithi115.stdout: rgw cache enabled = true 2024-09-18T01:16:29.265 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable ops log = true 2024-09-18T01:16:29.265 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable usage log = true 2024-09-18T01:16:29.265 INFO:teuthology.orchestra.run.smithi115.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T01:16:29.265 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T01:16:29.266 INFO:teuthology.orchestra.run.smithi115.stdout: debug rbd = 20 2024-09-18T01:16:29.266 INFO:teuthology.orchestra.run.smithi115.stdout: debug rbd_mirror = 15 2024-09-18T01:16:29.266 INFO:teuthology.orchestra.run.smithi115.stdout: log to stderr = False 2024-09-18T01:16:29.266 INFO:teuthology.orchestra.run.smithi115.stdout: rbd default features = 125 2024-09-18T01:16:29.266 INFO:teuthology.orchestra.run.smithi115.stdout: rbd mirror image policy type = none 2024-09-18T01:16:29.266 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.a] 2024-09-18T01:16:29.266 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.0] 2024-09-18T01:16:29.267 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:29.267 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:29.267 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.1] 2024-09-18T01:16:29.267 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:29.267 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:29.267 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.2] 2024-09-18T01:16:29.267 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:29.268 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:29.268 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.3] 2024-09-18T01:16:29.268 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-18T01:16:29.268 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-18T01:16:29.269 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-18T01:16:29.270 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-18T01:16:29.364 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T01:16:29.364 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:29.364 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-18T01:16:29.380 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:29.380 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-18T01:16:29.435 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:16:29.436 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:29.436 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-18T01:16:29.436 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-18T01:16:29.477 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:29.477 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-18T01:16:29.535 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:16:29.535 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:29.535 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-18T01:16:29.535 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-18T01:16:29.570 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:29.570 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-18T01:16:29.626 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T01:16:29.626 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T01:16:29.626 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-18T01:16:29.685 INFO:teuthology.orchestra.run.smithi066.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-18T01:16:29.688 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T01:16:29.688 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T01:16:29.688 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-18T01:16:29.733 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-18T01:16:29.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-18T01:16:29.787 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-18T01:16:29.800 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-18T01:16:29.881 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-18T01:16:29.893 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-18T01:16:29.974 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-18T01:16:29.987 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-18T01:16:30.067 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-18T01:16:30.080 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-18T01:16:30.160 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-18T01:16:30.173 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-18T01:16:30.253 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-18T01:16:30.269 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-18T01:16:30.349 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-18T01:16:30.362 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-18T01:16:30.442 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-18T01:16:30.454 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T01:16:30.455 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-18T01:16:30.455 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-18T01:16:30.519 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T01:16:30.520 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-18T01:16:30.520 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:16:30.520 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T01:16:30.592 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T01:16:31.100 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2024-09-18T01:16:31.102 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2024-09-18T01:16:31.102 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-18T01:16:31.242 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-18T01:16:31.274 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-18T01:16:31.340 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T01:16:31.340 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-18T01:16:31.341 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:16:31.341 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T01:16:31.413 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T01:16:31.413 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T01:16:31.413 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T01:16:31.414 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T01:16:31.414 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T01:16:31.414 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2024-09-18T01:16:31.414 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T01:16:31.414 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T01:16:31.414 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T01:16:31.414 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T01:16:31.871 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2024-09-18T01:16:31.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2024-09-18T01:16:31.873 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-18T01:16:31.964 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-18T01:16:31.993 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-18T01:16:32.059 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T01:16:32.059 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-18T01:16:32.059 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:16:32.059 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T01:16:32.133 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T01:16:32.134 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T01:16:32.625 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2024-09-18T01:16:32.626 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2024-09-18T01:16:32.627 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-18T01:16:32.757 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-18T01:16:32.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-18T01:16:32.893 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:32.892+0000 7f81a200f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-18T01:16:32.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:32.898+0000 7f81a200f640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-18T01:16:32.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:32.898+0000 7f81a200f640 -1 bdev(0x55dc28397800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-18T01:16:32.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:32.898+0000 7f81a200f640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-18T01:16:35.819 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-18T01:16:35.848 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-18T01:16:35.938 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:35.937+0000 7fb2d5c24640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-18T01:16:35.939 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:35.938+0000 7fb2d5c24640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-18T01:16:35.939 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:35.938+0000 7fb2d5c24640 -1 bdev(0x55b728215800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-18T01:16:35.939 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:35.938+0000 7fb2d5c24640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-18T01:16:39.100 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-18T01:16:39.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-18T01:16:39.215 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:39.213+0000 7ff71640c640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-18T01:16:39.215 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:39.213+0000 7ff71640c640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-18T01:16:39.216 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:39.213+0000 7ff71640c640 -1 bdev(0x5612a5701800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-18T01:16:39.216 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:16:39.213+0000 7ff71640c640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-18T01:16:42.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-18T01:16:42.155 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T01:16:42.155 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:42.155 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-18T01:16:42.187 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.187 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-18T01:16:42.224 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.224 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-18T01:16:42.292 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.292 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-18T01:16:42.359 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-09-18T01:16:42.359 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-18T01:16:42.378 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.379 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-18T01:16:42.416 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.416 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-18T01:16:42.473 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.473 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-18T01:16:42.529 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.530 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-18T01:16:42.586 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.586 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-18T01:16:42.641 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.641 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-18T01:16:42.697 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.697 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-18T01:16:42.754 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:42.754 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-18T01:16:42.809 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T01:16:42.810 DEBUG:teuthology.orchestra.run.smithi115:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.x] 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtKepmhaLdKBAA+c7XakL4gPYT7Fz6gNs1ng== 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.0] 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDwKepmXfBCNRAAJoJUgCSsXsE29TNlk6QERA== 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.1] 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDzKepmhHL0NxAAtMVhpbP9GsC9Ne2RvV8gbQ== 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.2] 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQD3KepmPyfUDBAAQYLfwgTxdsOTEfyCuiDVkA== 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtKepm+Ba5KxAAczyfIUvLlod6/lRuEQrw2g== 2024-09-18T01:16:42.874 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror] 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtKepmzSz2LhAAqHsTMCop6oQ5HA3cvE5vuA== 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.0] 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtKepmyGWGNBAAiisN+HzoNhi6+2Z4ys9SSw== 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.1] 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtKepmod0XOhAAd6OYS2j48E4BT/2UGgvUVw== 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.2] 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDuKepm2TEJBBAAmnsLZEmxjzJs8UBklBE0aw== 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.3] 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDuKepmR9qSCRAAvJn0NfQH1gK1V3M57YX5Hg== 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.4] 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDuKepmBUAcDxAA9RLQ6km1izebj4gqyj9oIQ== 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.5] 2024-09-18T01:16:42.875 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDuKepm2sDWFBAAdZxvKmSqnEcKM4yvCekG4Q== 2024-09-18T01:16:42.876 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.6] 2024-09-18T01:16:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDuKepmIbhiGhAAWVq7vod4FALCTXna21OjZQ== 2024-09-18T01:16:42.877 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T01:16:42.970 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T01:16:43.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T01:16:43.072 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T01:16:43.122 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.229 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.283 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.336 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.390 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.443 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.497 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.550 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T01:16:43.603 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T01:16:43.604 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-18T01:16:43.628 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-18T01:16:43.722 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-18T01:16:43.747 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-18T01:16:43.802 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-18T01:16:43.802 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-18T01:16:43.803 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-18T01:16:43.845 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-18T01:16:43.845 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-18T01:16:43.845 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-18T01:16:43.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-18T01:16:43.848 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-18T01:16:43.848 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T01:16:43.848 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-18T01:16:44.016 INFO:teuthology.orchestra.run.smithi115.stderr:adjusted tunables profile to default 2024-09-18T01:16:44.018 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.018+0000 7f59aa528040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T01:16:44.027 INFO:tasks.ceph:check_enable_crimson: False 2024-09-18T01:16:44.027 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-18T01:16:44.028 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:44.028 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-18T01:16:44.052 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:44.052 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-18T01:16:44.061 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.061+0000 7f59aa528040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T01:16:44.117 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:16:44.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-18T01:16:44.182 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster cluster2 osd new f1629294-c19f-4008-ad1d-71ca6c8aa76e 0 2024-09-18T01:16:44.198 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.198+0000 7f59aa528040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-18T01:16:44.248 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.248+0000 7f59aa528040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T01:16:44.361 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.361+0000 7f59aa528040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T01:16:44.394 INFO:teuthology.orchestra.run.smithi115.stdout:0 2024-09-18T01:16:44.404 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster cluster2 osd new 0b9f55f6-b049-401f-9d1a-9a7ff1167146 1 2024-09-18T01:16:44.458 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.458+0000 7f59aa528040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T01:16:44.577 INFO:teuthology.orchestra.run.smithi115.stdout:1 2024-09-18T01:16:44.588 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster cluster2 osd new 0b29db8a-24e4-49bc-aac8-dbf6b6f2fa69 2 2024-09-18T01:16:44.732 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.731+0000 7f59aa528040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T01:16:44.760 INFO:teuthology.orchestra.run.smithi115.stdout:2 2024-09-18T01:16:44.771 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-18T01:16:44.771 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-18T01:16:44.773 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-18T01:16:44.774 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-18T01:16:44.774 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-18T01:16:44.776 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-18T01:16:44.776 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-18T01:16:44.776 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-18T01:16:44.777 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.777+0000 7f59aa528040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T01:16:44.778 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-18T01:16:44.778 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-18T01:16:44.899 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:44.898+0000 7f59aa528040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T01:16:44.946 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:16:44.946 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":5,"fsid":"fcf9d702-6ece-4878-bfd9-543b84db5aa5","created":"2024-09-18T01:16:43.930763+0000","modified":"2024-09-18T01:16:44.760042+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T01:16:44.760042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f1629294-c19f-4008-ad1d-71ca6c8aa76e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0b9f55f6-b049-401f-9d1a-9a7ff1167146","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0b29db8a-24e4-49bc-aac8-dbf6b6f2fa69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:44.957 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-18T01:16:44.958 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T01:16:45.260 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-18T01:16:45.417 INFO:teuthology.misc.health.smithi115.stdout: 2024-09-18T01:16:45.418 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":5,"fsid":"fcf9d702-6ece-4878-bfd9-543b84db5aa5","created":"2024-09-18T01:16:43.930763+0000","modified":"2024-09-18T01:16:44.760042+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T01:16:44.760042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f1629294-c19f-4008-ad1d-71ca6c8aa76e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0b9f55f6-b049-401f-9d1a-9a7ff1167146","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0b29db8a-24e4-49bc-aac8-dbf6b6f2fa69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:45.427 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-18T01:16:45.624 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:45.623+0000 7f59aa528040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T01:16:45.669 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:45.669+0000 7f59aa528040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T01:16:45.974 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:16:45.972+0000 7f790ae0b640 -1 Falling back to public interface 2024-09-18T01:16:45.996 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:45.996+0000 7f59aa528040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-18T01:16:46.083 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.083+0000 7f59aa528040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T01:16:46.194 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.194+0000 7f59aa528040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T01:16:46.224 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:16:46.223+0000 7fade7618640 -1 Falling back to public interface 2024-09-18T01:16:46.238 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.238+0000 7f59aa528040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T01:16:46.353 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.353+0000 7f59aa528040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-18T01:16:46.407 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.406+0000 7f59aa528040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T01:16:46.436 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:16:46.435+0000 7f67a0816640 -1 Falling back to public interface 2024-09-18T01:16:46.450 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.449+0000 7f59aa528040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T01:16:46.566 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.566+0000 7f59aa528040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T01:16:46.616 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.616+0000 7f59aa528040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T01:16:46.666 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.666+0000 7f59aa528040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T01:16:46.721 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.720+0000 7f59aa528040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T01:16:46.764 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.764+0000 7f59aa528040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T01:16:46.808 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:46.807+0000 7f59aa528040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T01:16:47.395 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:47.395+0000 7f59aa528040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T01:16:48.462 INFO:tasks.ceph.cluster2.mgr.x.smithi066.stderr:2024-09-18T01:16:48.461+0000 7f594a518640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-18T01:16:49.754 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:16:49.753+0000 7f790ae0b640 -1 osd.2 0 log_to_monitors true 2024-09-18T01:16:50.193 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:16:50.192+0000 7f67a0816640 -1 osd.1 0 log_to_monitors true 2024-09-18T01:16:50.494 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:16:50.493+0000 7fade7618640 -1 osd.0 0 log_to_monitors true 2024-09-18T01:16:51.731 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-18T01:16:52.119 INFO:teuthology.misc.health.smithi115.stdout: 2024-09-18T01:16:52.119 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":7,"fsid":"fcf9d702-6ece-4878-bfd9-543b84db5aa5","created":"2024-09-18T01:16:43.930763+0000","modified":"2024-09-18T01:16:51.460938+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T01:16:44.760042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f1629294-c19f-4008-ad1d-71ca6c8aa76e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0b9f55f6-b049-401f-9d1a-9a7ff1167146","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0b29db8a-24e4-49bc-aac8-dbf6b6f2fa69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:52.133 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-18T01:16:52.362 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:16:52.361+0000 7f7906831640 -1 osd.2 0 waiting for initial osdmap 2024-09-18T01:16:52.377 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:16:52.376+0000 7f78fe651640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T01:16:52.411 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:16:52.411+0000 7f679c17a640 -1 osd.1 0 waiting for initial osdmap 2024-09-18T01:16:52.426 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:16:52.426+0000 7f6793f9a640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T01:16:53.153 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:16:53.152+0000 7fade2e9f640 -1 osd.0 0 waiting for initial osdmap 2024-09-18T01:16:53.167 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:16:53.166+0000 7faddacbf640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T01:16:58.436 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-18T01:16:58.718 INFO:teuthology.misc.health.smithi115.stdout: 2024-09-18T01:16:58.718 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":12,"fsid":"fcf9d702-6ece-4878-bfd9-543b84db5aa5","created":"2024-09-18T01:16:43.930763+0000","modified":"2024-09-18T01:16:56.482931+0000","last_up_change":"2024-09-18T01:16:53.471910+0000","last_in_change":"2024-09-18T01:16:44.760042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T01:16:53.632560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1629294-c19f-4008-ad1d-71ca6c8aa76e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6809","nonce":2571597248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6811","nonce":2571597248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6815","nonce":2571597248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6813","nonce":2571597248}]},"public_addr":"172.21.15.115:6809/2571597248","cluster_addr":"172.21.15.115:6811/2571597248","heartbeat_back_addr":"172.21.15.115:6815/2571597248","heartbeat_front_addr":"172.21.15.115:6813/2571597248","state":["exists","up"]},{"osd":1,"uuid":"0b9f55f6-b049-401f-9d1a-9a7ff1167146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6817","nonce":1967664521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6819","nonce":1967664521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6823","nonce":1967664521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6821","nonce":1967664521}]},"public_addr":"172.21.15.115:6817/1967664521","cluster_addr":"172.21.15.115:6819/1967664521","heartbeat_back_addr":"172.21.15.115:6823/1967664521","heartbeat_front_addr":"172.21.15.115:6821/1967664521","state":["exists","up"]},{"osd":2,"uuid":"0b29db8a-24e4-49bc-aac8-dbf6b6f2fa69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6801","nonce":1836880245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6803","nonce":1836880245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6807","nonce":1836880245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6805","nonce":1836880245}]},"public_addr":"172.21.15.115:6801/1836880245","cluster_addr":"172.21.15.115:6803/1836880245","heartbeat_back_addr":"172.21.15.115:6807/1836880245","heartbeat_front_addr":"172.21.15.115:6805/1836880245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:51.445385+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:51.218875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:50.713702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:16:58.729 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-18T01:16:58.729 INFO:tasks.ceph:Creating RBD pool 2024-09-18T01:16:58.729 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-18T01:16:59.495 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'rbd' created 2024-09-18T01:16:59.510 DEBUG:teuthology.orchestra.run.smithi115:> rbd --cluster cluster2 pool init rbd 2024-09-18T01:17:02.520 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-18T01:17:02.521 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-18T01:17:02.521 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-18T01:17:02.521 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-18T01:17:02.521 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T01:17:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:17:02.861 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":4294627999},{"type":"v1","addr":"172.21.15.66:6827","nonce":4294627999}]},"active_addr":"172.21.15.66:6827/4294627999","active_change":"2024-09-18T01:16:47.397227+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":1837835347}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":3735607575}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":2816161206}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":1489377286}]}]} 2024-09-18T01:17:02.863 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-18T01:17:02.863 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-18T01:17:02.863 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-18T01:17:03.138 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:17:03.138 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":16,"fsid":"fcf9d702-6ece-4878-bfd9-543b84db5aa5","created":"2024-09-18T01:16:43.930763+0000","modified":"2024-09-18T01:17:02.507885+0000","last_up_change":"2024-09-18T01:16:53.471910+0000","last_in_change":"2024-09-18T01:16:44.760042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T01:16:53.632560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T01:16:59.021351+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1629294-c19f-4008-ad1d-71ca6c8aa76e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6809","nonce":2571597248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6811","nonce":2571597248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6815","nonce":2571597248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6813","nonce":2571597248}]},"public_addr":"172.21.15.115:6809/2571597248","cluster_addr":"172.21.15.115:6811/2571597248","heartbeat_back_addr":"172.21.15.115:6815/2571597248","heartbeat_front_addr":"172.21.15.115:6813/2571597248","state":["exists","up"]},{"osd":1,"uuid":"0b9f55f6-b049-401f-9d1a-9a7ff1167146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6817","nonce":1967664521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6819","nonce":1967664521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6823","nonce":1967664521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6821","nonce":1967664521}]},"public_addr":"172.21.15.115:6817/1967664521","cluster_addr":"172.21.15.115:6819/1967664521","heartbeat_back_addr":"172.21.15.115:6823/1967664521","heartbeat_front_addr":"172.21.15.115:6821/1967664521","state":["exists","up"]},{"osd":2,"uuid":"0b29db8a-24e4-49bc-aac8-dbf6b6f2fa69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6801","nonce":1836880245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6803","nonce":1836880245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6807","nonce":1836880245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6805","nonce":1836880245}]},"public_addr":"172.21.15.115:6801/1836880245","cluster_addr":"172.21.15.115:6803/1836880245","heartbeat_back_addr":"172.21.15.115:6807/1836880245","heartbeat_front_addr":"172.21.15.115:6805/1836880245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:51.445385+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:51.218875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:50.713702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:17:03.151 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-18T01:17:03.151 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-18T01:17:03.425 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:17:03.425 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":16,"fsid":"fcf9d702-6ece-4878-bfd9-543b84db5aa5","created":"2024-09-18T01:16:43.930763+0000","modified":"2024-09-18T01:17:02.507885+0000","last_up_change":"2024-09-18T01:16:53.471910+0000","last_in_change":"2024-09-18T01:16:44.760042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T01:16:53.632560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T01:16:59.021351+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1629294-c19f-4008-ad1d-71ca6c8aa76e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6809","nonce":2571597248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6811","nonce":2571597248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6815","nonce":2571597248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6813","nonce":2571597248}]},"public_addr":"172.21.15.115:6809/2571597248","cluster_addr":"172.21.15.115:6811/2571597248","heartbeat_back_addr":"172.21.15.115:6815/2571597248","heartbeat_front_addr":"172.21.15.115:6813/2571597248","state":["exists","up"]},{"osd":1,"uuid":"0b9f55f6-b049-401f-9d1a-9a7ff1167146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6817","nonce":1967664521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6819","nonce":1967664521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6823","nonce":1967664521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6821","nonce":1967664521}]},"public_addr":"172.21.15.115:6817/1967664521","cluster_addr":"172.21.15.115:6819/1967664521","heartbeat_back_addr":"172.21.15.115:6823/1967664521","heartbeat_front_addr":"172.21.15.115:6821/1967664521","state":["exists","up"]},{"osd":2,"uuid":"0b29db8a-24e4-49bc-aac8-dbf6b6f2fa69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6801","nonce":1836880245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6803","nonce":1836880245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6807","nonce":1836880245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6805","nonce":1836880245}]},"public_addr":"172.21.15.115:6801/1836880245","cluster_addr":"172.21.15.115:6803/1836880245","heartbeat_back_addr":"172.21.15.115:6807/1836880245","heartbeat_front_addr":"172.21.15.115:6805/1836880245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:51.445385+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:51.218875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:50.713702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:17:03.437 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-18T01:17:03.438 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-18T01:17:03.438 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-18T01:17:03.577 INFO:teuthology.orchestra.run.smithi115.stdout:34359738374 2024-09-18T01:17:03.577 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-18T01:17:03.580 INFO:teuthology.orchestra.run.smithi115.stdout:34359738374 2024-09-18T01:17:03.581 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-18T01:17:03.581 INFO:teuthology.orchestra.run.smithi115.stdout:38654705669 2024-09-18T01:17:03.581 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-18T01:17:03.888 INFO:teuthology.orchestra.run.smithi115.stdout:34359738372 2024-09-18T01:17:03.900 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.2 2024-09-18T01:17:03.911 INFO:teuthology.orchestra.run.smithi115.stdout:34359738372 2024-09-18T01:17:03.924 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1 2024-09-18T01:17:03.931 INFO:teuthology.orchestra.run.smithi115.stdout:38654705667 2024-09-18T01:17:03.944 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705667 for osd.0 2024-09-18T01:17:04.901 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-18T01:17:04.925 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-18T01:17:04.944 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-18T01:17:05.237 INFO:teuthology.orchestra.run.smithi115.stdout:38654705667 2024-09-18T01:17:05.241 INFO:teuthology.orchestra.run.smithi115.stdout:34359738372 2024-09-18T01:17:05.250 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705667 for osd.0 2024-09-18T01:17:05.255 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.2 2024-09-18T01:17:05.270 INFO:teuthology.orchestra.run.smithi115.stdout:34359738372 2024-09-18T01:17:05.283 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1 2024-09-18T01:17:06.251 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-18T01:17:06.256 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-18T01:17:06.284 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-18T01:17:06.557 INFO:teuthology.orchestra.run.smithi115.stdout:38654705669 2024-09-18T01:17:06.570 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705669 for osd.0 2024-09-18T01:17:06.570 DEBUG:teuthology.parallel:result is None 2024-09-18T01:17:06.588 INFO:teuthology.orchestra.run.smithi115.stdout:34359738374 2024-09-18T01:17:06.594 INFO:teuthology.orchestra.run.smithi115.stdout:34359738374 2024-09-18T01:17:06.601 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-09-18T01:17:06.602 DEBUG:teuthology.parallel:result is None 2024-09-18T01:17:06.607 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2024-09-18T01:17:06.607 DEBUG:teuthology.parallel:result is None 2024-09-18T01:17:06.608 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-18T01:17:06.608 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T01:17:06.920 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:17:06.920 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-18T01:17:06.932 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-18T01:17:05.404108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81628,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033892,"statfs":{"total":289910292480,"available":289826705408,"internally_reserved":0,"allocated":548864,"data_stored":1304053,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.905441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518768+0000","last_change":"2024-09-18T01:17:02.518768+0000","last_active":"2024-09-18T01:17:02.518768+0000","last_peered":"2024-09-18T01:17:02.518768+0000","last_clean":"2024-09-18T01:17:02.518768+0000","last_became_active":"2024-09-18T01:17:00.511441+0000","last_became_peered":"2024-09-18T01:17:00.511441+0000","last_unstale":"2024-09-18T01:17:02.518768+0000","last_undegraded":"2024-09-18T01:17:02.518768+0000","last_fullsized":"2024-09-18T01:17:02.518768+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:23:40.338890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000687489,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518477+0000","last_change":"2024-09-18T01:17:02.518477+0000","last_active":"2024-09-18T01:17:02.518477+0000","last_peered":"2024-09-18T01:17:02.518477+0000","last_clean":"2024-09-18T01:17:02.518477+0000","last_became_active":"2024-09-18T01:17:00.510105+0000","last_became_peered":"2024-09-18T01:17:00.510105+0000","last_unstale":"2024-09-18T01:17:02.518477+0000","last_undegraded":"2024-09-18T01:17:02.518477+0000","last_fullsized":"2024-09-18T01:17:02.518477+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:12:46.477909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043520000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.519127+0000","last_change":"2024-09-18T01:17:02.519127+0000","last_active":"2024-09-18T01:17:02.519127+0000","last_peered":"2024-09-18T01:17:02.519127+0000","last_clean":"2024-09-18T01:17:02.519127+0000","last_became_active":"2024-09-18T01:17:00.509844+0000","last_became_peered":"2024-09-18T01:17:00.509844+0000","last_unstale":"2024-09-18T01:17:02.519127+0000","last_undegraded":"2024-09-18T01:17:02.519127+0000","last_fullsized":"2024-09-18T01:17:02.519127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:45:37.121984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518480+0000","last_change":"2024-09-18T01:17:02.518480+0000","last_active":"2024-09-18T01:17:02.518480+0000","last_peered":"2024-09-18T01:17:02.518480+0000","last_clean":"2024-09-18T01:17:02.518480+0000","last_became_active":"2024-09-18T01:17:00.509856+0000","last_became_peered":"2024-09-18T01:17:00.509856+0000","last_unstale":"2024-09-18T01:17:02.518480+0000","last_undegraded":"2024-09-18T01:17:02.518480+0000","last_fullsized":"2024-09-18T01:17:02.518480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:34:28.368843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042899300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.514858+0000","last_change":"2024-09-18T01:17:02.513682+0000","last_active":"2024-09-18T01:17:02.514858+0000","last_peered":"2024-09-18T01:17:02.514858+0000","last_clean":"2024-09-18T01:17:02.514858+0000","last_became_active":"2024-09-18T01:17:00.511011+0000","last_became_peered":"2024-09-18T01:17:00.511011+0000","last_unstale":"2024-09-18T01:17:02.514858+0000","last_undegraded":"2024-09-18T01:17:02.514858+0000","last_fullsized":"2024-09-18T01:17:02.514858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:14:05.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011029220000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.511032+0000","last_change":"2024-09-18T01:17:02.511032+0000","last_active":"2024-09-18T01:17:02.511032+0000","last_peered":"2024-09-18T01:17:02.511032+0000","last_clean":"2024-09-18T01:17:02.511032+0000","last_became_active":"2024-09-18T01:17:00.510439+0000","last_became_peered":"2024-09-18T01:17:00.510439+0000","last_unstale":"2024-09-18T01:17:02.511032+0000","last_undegraded":"2024-09-18T01:17:02.511032+0000","last_fullsized":"2024-09-18T01:17:02.511032+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:04:11.663327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.510963+0000","last_change":"2024-09-18T01:17:02.510963+0000","last_active":"2024-09-18T01:17:02.510963+0000","last_peered":"2024-09-18T01:17:02.510963+0000","last_clean":"2024-09-18T01:17:02.510963+0000","last_became_active":"2024-09-18T01:17:00.512356+0000","last_became_peered":"2024-09-18T01:17:00.512356+0000","last_unstale":"2024-09-18T01:17:02.510963+0000","last_undegraded":"2024-09-18T01:17:02.510963+0000","last_fullsized":"2024-09-18T01:17:02.510963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:42:49.393044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518778+0000","last_change":"2024-09-18T01:17:02.518778+0000","last_active":"2024-09-18T01:17:02.518778+0000","last_peered":"2024-09-18T01:17:02.518778+0000","last_clean":"2024-09-18T01:17:02.518778+0000","last_became_active":"2024-09-18T01:17:00.509327+0000","last_became_peered":"2024-09-18T01:17:00.509327+0000","last_unstale":"2024-09-18T01:17:02.518778+0000","last_undegraded":"2024-09-18T01:17:02.518778+0000","last_fullsized":"2024-09-18T01:17:02.518778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:26:58.168836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069378899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518064+0000","last_change":"2024-09-18T01:16:55.498135+0000","last_active":"2024-09-18T01:17:02.518064+0000","last_peered":"2024-09-18T01:17:02.518064+0000","last_clean":"2024-09-18T01:17:02.518064+0000","last_became_active":"2024-09-18T01:16:55.497021+0000","last_became_peered":"2024-09-18T01:16:55.497021+0000","last_unstale":"2024-09-18T01:17:02.518064+0000","last_undegraded":"2024-09-18T01:17:02.518064+0000","last_fullsized":"2024-09-18T01:17:02.518064+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_clean_scrub_stamp":"2024-09-18T01:16:54.479487+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:24:07.471201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":41093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":204800,"data_stored":631480,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":631480,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:17:06.933 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T01:17:07.202 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:17:07.202 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-18T01:17:07.213 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-18T01:17:05.404108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81628,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033892,"statfs":{"total":289910292480,"available":289826705408,"internally_reserved":0,"allocated":548864,"data_stored":1304053,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.905441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518768+0000","last_change":"2024-09-18T01:17:02.518768+0000","last_active":"2024-09-18T01:17:02.518768+0000","last_peered":"2024-09-18T01:17:02.518768+0000","last_clean":"2024-09-18T01:17:02.518768+0000","last_became_active":"2024-09-18T01:17:00.511441+0000","last_became_peered":"2024-09-18T01:17:00.511441+0000","last_unstale":"2024-09-18T01:17:02.518768+0000","last_undegraded":"2024-09-18T01:17:02.518768+0000","last_fullsized":"2024-09-18T01:17:02.518768+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:23:40.338890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000687489,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518477+0000","last_change":"2024-09-18T01:17:02.518477+0000","last_active":"2024-09-18T01:17:02.518477+0000","last_peered":"2024-09-18T01:17:02.518477+0000","last_clean":"2024-09-18T01:17:02.518477+0000","last_became_active":"2024-09-18T01:17:00.510105+0000","last_became_peered":"2024-09-18T01:17:00.510105+0000","last_unstale":"2024-09-18T01:17:02.518477+0000","last_undegraded":"2024-09-18T01:17:02.518477+0000","last_fullsized":"2024-09-18T01:17:02.518477+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:12:46.477909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043520000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.519127+0000","last_change":"2024-09-18T01:17:02.519127+0000","last_active":"2024-09-18T01:17:02.519127+0000","last_peered":"2024-09-18T01:17:02.519127+0000","last_clean":"2024-09-18T01:17:02.519127+0000","last_became_active":"2024-09-18T01:17:00.509844+0000","last_became_peered":"2024-09-18T01:17:00.509844+0000","last_unstale":"2024-09-18T01:17:02.519127+0000","last_undegraded":"2024-09-18T01:17:02.519127+0000","last_fullsized":"2024-09-18T01:17:02.519127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:45:37.121984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518480+0000","last_change":"2024-09-18T01:17:02.518480+0000","last_active":"2024-09-18T01:17:02.518480+0000","last_peered":"2024-09-18T01:17:02.518480+0000","last_clean":"2024-09-18T01:17:02.518480+0000","last_became_active":"2024-09-18T01:17:00.509856+0000","last_became_peered":"2024-09-18T01:17:00.509856+0000","last_unstale":"2024-09-18T01:17:02.518480+0000","last_undegraded":"2024-09-18T01:17:02.518480+0000","last_fullsized":"2024-09-18T01:17:02.518480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:34:28.368843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042899300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.514858+0000","last_change":"2024-09-18T01:17:02.513682+0000","last_active":"2024-09-18T01:17:02.514858+0000","last_peered":"2024-09-18T01:17:02.514858+0000","last_clean":"2024-09-18T01:17:02.514858+0000","last_became_active":"2024-09-18T01:17:00.511011+0000","last_became_peered":"2024-09-18T01:17:00.511011+0000","last_unstale":"2024-09-18T01:17:02.514858+0000","last_undegraded":"2024-09-18T01:17:02.514858+0000","last_fullsized":"2024-09-18T01:17:02.514858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:14:05.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011029220000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.511032+0000","last_change":"2024-09-18T01:17:02.511032+0000","last_active":"2024-09-18T01:17:02.511032+0000","last_peered":"2024-09-18T01:17:02.511032+0000","last_clean":"2024-09-18T01:17:02.511032+0000","last_became_active":"2024-09-18T01:17:00.510439+0000","last_became_peered":"2024-09-18T01:17:00.510439+0000","last_unstale":"2024-09-18T01:17:02.511032+0000","last_undegraded":"2024-09-18T01:17:02.511032+0000","last_fullsized":"2024-09-18T01:17:02.511032+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:04:11.663327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.510963+0000","last_change":"2024-09-18T01:17:02.510963+0000","last_active":"2024-09-18T01:17:02.510963+0000","last_peered":"2024-09-18T01:17:02.510963+0000","last_clean":"2024-09-18T01:17:02.510963+0000","last_became_active":"2024-09-18T01:17:00.512356+0000","last_became_peered":"2024-09-18T01:17:00.512356+0000","last_unstale":"2024-09-18T01:17:02.510963+0000","last_undegraded":"2024-09-18T01:17:02.510963+0000","last_fullsized":"2024-09-18T01:17:02.510963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:42:49.393044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518778+0000","last_change":"2024-09-18T01:17:02.518778+0000","last_active":"2024-09-18T01:17:02.518778+0000","last_peered":"2024-09-18T01:17:02.518778+0000","last_clean":"2024-09-18T01:17:02.518778+0000","last_became_active":"2024-09-18T01:17:00.509327+0000","last_became_peered":"2024-09-18T01:17:00.509327+0000","last_unstale":"2024-09-18T01:17:02.518778+0000","last_undegraded":"2024-09-18T01:17:02.518778+0000","last_fullsized":"2024-09-18T01:17:02.518778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:26:58.168836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069378899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T01:17:02.518064+0000","last_change":"2024-09-18T01:16:55.498135+0000","last_active":"2024-09-18T01:17:02.518064+0000","last_peered":"2024-09-18T01:17:02.518064+0000","last_clean":"2024-09-18T01:17:02.518064+0000","last_became_active":"2024-09-18T01:16:55.497021+0000","last_became_peered":"2024-09-18T01:16:55.497021+0000","last_unstale":"2024-09-18T01:17:02.518064+0000","last_undegraded":"2024-09-18T01:17:02.518064+0000","last_fullsized":"2024-09-18T01:17:02.518064+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_clean_scrub_stamp":"2024-09-18T01:16:54.479487+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:24:07.471201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":41093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":204800,"data_stored":631480,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":631480,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:17:07.214 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-18T01:17:07.214 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-18T01:17:07.214 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-18T01:17:07.215 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-18T01:17:07.526 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:17:07.527 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T01:17:07.539 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-18T01:17:07.540 INFO:teuthology.run_tasks:Running task exec... 2024-09-18T01:17:07.548 INFO:teuthology.task.exec:Executing custom commands... 2024-09-18T01:17:07.549 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:17:07.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:07.896 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror] 2024-09-18T01:17:07.897 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDHKepm7KQiFxAA1f8smTcAJRy5aU2QQRA+dg== 2024-09-18T01:17:07.897 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-18T01:17:07.897 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:07.897 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror 2024-09-18T01:17:07.910 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:08.256 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.0] 2024-09-18T01:17:08.256 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDHKepm3+JuGhAA4d9ZTJcdVGecnU6J+O3v2A== 2024-09-18T01:17:08.256 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror" 2024-09-18T01:17:08.257 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:08.257 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror.0 2024-09-18T01:17:08.270 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:08.619 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.1] 2024-09-18T01:17:08.619 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDHKepmgj38HxAAbsBzpXOLd0n9uSTUrdicVQ== 2024-09-18T01:17:08.619 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror" 2024-09-18T01:17:08.619 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:08.620 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror.1 2024-09-18T01:17:08.633 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:08.983 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.2] 2024-09-18T01:17:08.983 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDHKepmVeKLJRAAAx1nRW8W0e0s8UqgohXdZw== 2024-09-18T01:17:08.983 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror" 2024-09-18T01:17:08.983 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:08.984 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror.2 2024-09-18T01:17:08.996 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:09.344 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.3] 2024-09-18T01:17:09.344 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDHKepmej8uKxAAJN/ePMQ64SN25RdB064RVQ== 2024-09-18T01:17:09.344 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror" 2024-09-18T01:17:09.344 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:09.344 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror.3 2024-09-18T01:17:09.357 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:09.705 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror] 2024-09-18T01:17:09.706 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtKepmzSz2LhAAqHsTMCop6oQ5HA3cvE5vuA== 2024-09-18T01:17:09.706 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-18T01:17:09.706 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:09.706 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror 2024-09-18T01:17:09.719 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:10.067 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.0] 2024-09-18T01:17:10.067 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtKepmyGWGNBAAiisN+HzoNhi6+2Z4ys9SSw== 2024-09-18T01:17:10.068 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror" 2024-09-18T01:17:10.068 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:10.068 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror.0 2024-09-18T01:17:10.081 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.1] 2024-09-18T01:17:10.427 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtKepmod0XOhAAd6OYS2j48E4BT/2UGgvUVw== 2024-09-18T01:17:10.427 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror" 2024-09-18T01:17:10.427 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:10.427 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror.1 2024-09-18T01:17:10.440 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:10.788 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.2] 2024-09-18T01:17:10.788 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDuKepm2TEJBBAAmnsLZEmxjzJs8UBklBE0aw== 2024-09-18T01:17:10.788 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror" 2024-09-18T01:17:10.788 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:10.788 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror.2 2024-09-18T01:17:10.801 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-18T01:17:11.147 INFO:teuthology.orchestra.run.smithi115.stdout:[client.mirror.3] 2024-09-18T01:17:11.147 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDuKepmR9qSCRAAvJn0NfQH1gK1V3M57YX5Hg== 2024-09-18T01:17:11.147 INFO:teuthology.orchestra.run.smithi115.stdout: caps mon = "profile rbd-mirror" 2024-09-18T01:17:11.147 INFO:teuthology.orchestra.run.smithi115.stdout: caps osd = "profile rbd" 2024-09-18T01:17:11.147 INFO:teuthology.orchestra.run.smithi115.stderr:updated caps for client.mirror.3 2024-09-18T01:17:11.161 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-18T01:17:11.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-18T01:17:11.178 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-18T01:17:11.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-18T01:17:11.181 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-18T01:17:11.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-18T01:17:11.332 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-18T01:17:11.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-18T01:17:11.335 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-18T01:17:11.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-18T01:17:11.403 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-18T01:17:11.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-18T01:17:11.406 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-18T01:17:11.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-18T01:17:11.499 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-18T01:17:11.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-18T01:17:11.502 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-18T01:17:11.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-18T01:17:11.513 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-18T01:17:11.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-18T01:17:11.515 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-18T01:17:11.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-18T01:17:11.526 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-18T01:17:11.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-18T01:17:11.528 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-18T01:17:11.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-18T01:17:11.541 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-18T01:17:11.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-18T01:17:11.544 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-18T01:17:11.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-18T01:17:11.553 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-18T01:17:11.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-18T01:17:11.556 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-18T01:17:11.659 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726622231 2024-09-18T01:17:11.659 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-18T01:17:11.660 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-18T01:17:11.671 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726622231 2024-09-18T01:17:11.671 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-18T01:17:11.671 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T01:17:11.689 INFO:tasks.workunit:Pulling workunits from ref b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 2024-09-18T01:17:11.690 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T01:17:11.690 DEBUG:teuthology.orchestra.run.smithi115:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-18T01:17:11.690 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-18T01:17:11.691 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-09-18T01:17:11.691 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-18T01:17:11.691 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-18T01:17:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:17:11.706 INFO:teuthology.orchestra.run.smithi115.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-18T01:17:11.706 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-18T01:17:11.762 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-18T01:17:11.763 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-18T01:17:11.817 INFO:tasks.workunit:timeout=6h 2024-09-18T01:17:11.817 INFO:tasks.workunit:cleanup=True 2024-09-18T01:17:11.818 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 2024-09-18T01:17:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-18T01:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Note: switching to 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'. 2024-09-18T01:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T01:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: git switch -c 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Or undo this operation with: 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:18:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: git switch - 2024-09-18T01:18:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:18:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T01:18:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:18:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:HEAD is now at b0d356b98e0 Merge pull request #59811 from afreen23/wip-68079-squid 2024-09-18T01:18:07.493 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-18T01:18:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-18T01:18:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-18T01:18:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-18T01:18:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-18T01:18:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-18T01:18:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-18T01:18:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-18T01:18:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-18T01:18:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-18T01:18:08.413 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T01:18:08.413 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-18T01:18:08.469 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-18T01:18:08.471 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-18T01:18:08.471 DEBUG:teuthology.orchestra.run.smithi115:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-18T01:18:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ IMAGE_COUNT=50 2024-09-18T01:18:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ export LOCKDEP=0 2024-09-18T01:18:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ LOCKDEP=0 2024-09-18T01:18:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ type xmlstarlet 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ XMLSTARLET=xmlstarlet 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CLUSTER1=cluster1 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CLUSTER2=cluster2 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ POOL=mirror 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ PARENT_POOL=mirror_parent 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ NS1=ns1 2024-09-18T01:18:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ NS2=ns2 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ TEMPDIR= 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ID=mirror 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ MIRROR_POOL_MODE=image 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' snapshot = snapshot ']' 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ MIRROR_POOL_MODE=image 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS='--id mirror' 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-18T01:18:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-18T01:18:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-18T01:18:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_BIN=. 2024-09-18T01:18:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_SRC=. 2024-09-18T01:18:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-18T01:18:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 -gt 0 ']' 2024-09-18T01:18:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ setup 2024-09-18T01:18:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local c 2024-09-18T01:18:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-18T01:18:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ setup_tempdir 2024-09-18T01:18:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mktemp -d 2024-09-18T01:18:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ TEMPDIR=/tmp/tmp.YxZgdY10x3 2024-09-18T01:18:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -z 1 ']' 2024-09-18T01:18:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ setup_pools cluster1 cluster2 2024-09-18T01:18:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:18:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local remote_cluster=cluster2 2024-09-18T01:18:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mon_map_file 2024-09-18T01:18:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mon_addr 2024-09-18T01:18:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local admin_key_file 2024-09-18T01:18:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local uuid 2024-09-18T01:18:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-18T01:18:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:pool 'mirror' already exists 2024-09-18T01:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-18T01:18:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:pool 'mirror_parent' already exists 2024-09-18T01:18:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-18T01:18:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-18T01:18:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-18T01:18:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-18T01:18:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-18T01:18:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-18T01:18:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-18T01:18:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-18T01:18:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -z ']' 2024-09-18T01:18:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -z ']' 2024-09-18T01:18:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ peer_add cluster1 mirror cluster2 2024-09-18T01:18:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:18:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local client_cluster=cluster2 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local remote_cluster=cluster2 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local uuid_var_name 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local error_code 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local peer_uuid 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 16 32 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ peer_uuid=cdfc9d9b-a45f-4747-afda-d9a066c547c9 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 17 ']' 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 -eq 0 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-18T01:18:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror_parent 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local client_cluster=cluster2 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local remote_cluster=cluster2 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local uuid_var_name 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local error_code 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local peer_uuid 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 16 32 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:18:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-18T01:18:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ peer_uuid=431fd1f2-2373-4f63-bca1-a43c51ab53bd 2024-09-18T01:18:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:18:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:18:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 17 ']' 2024-09-18T01:18:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 -eq 0 2024-09-18T01:18:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:18:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ setup_pools cluster2 cluster1 2024-09-18T01:18:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:18:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local remote_cluster=cluster1 2024-09-18T01:18:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mon_map_file 2024-09-18T01:18:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mon_addr 2024-09-18T01:18:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local admin_key_file 2024-09-18T01:18:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local uuid 2024-09-18T01:18:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-18T01:18:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:pool 'mirror' already exists 2024-09-18T01:18:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-18T01:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:pool 'mirror_parent' already exists 2024-09-18T01:18:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-18T01:18:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-18T01:18:24.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:24.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:18:24.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-18T01:18:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-18T01:18:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-18T01:18:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-18T01:18:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-18T01:18:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -z ']' 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -z ']' 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ peer_add cluster2 mirror cluster1 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local client_cluster=cluster1 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local remote_cluster=cluster1 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local uuid_var_name 2024-09-18T01:18:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local error_code 2024-09-18T01:18:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local peer_uuid 2024-09-18T01:18:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 16 32 2024-09-18T01:18:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:18:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-18T01:18:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ peer_uuid=4161be2f-bdf8-452e-8062-84f828027e86 2024-09-18T01:18:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:18:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:18:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 17 ']' 2024-09-18T01:18:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 -eq 0 2024-09-18T01:18:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror_parent 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local client_cluster=cluster1 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local remote_cluster=cluster1 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local uuid_var_name 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local error_code 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local peer_uuid 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 16 32 2024-09-18T01:18:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:18:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-18T01:18:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ peer_uuid=f9f60d49-2ca9-4de3-857b-4d997bfbf7c9 2024-09-18T01:18:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:18:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:18:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 17 ']' 2024-09-18T01:18:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 -eq 0 2024-09-18T01:18:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:18:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:18:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirrors cluster1 2024-09-18T01:18:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:18:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 0 3 2024-09-18T01:18:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster1:0 2024-09-18T01:18:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1:0 2024-09-18T01:18:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:18:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster1:0 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster1 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=0 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster1 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=0 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:18:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster1:1 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1:1 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster1:1 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster1 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=1 2024-09-18T01:18:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster1 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=1 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster1:2 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1:2 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster1:2 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:18:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster1 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=2 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster1 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=2 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster1:3 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1:3 2024-09-18T01:18:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster1:3 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster1 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=3 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster1 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=3 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:18:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirrors cluster2 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 0 3 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster2:0 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2:0 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster2:0 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:18:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster2 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=0 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster2 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster2 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=0 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:18:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster2:1 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2:1 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster2:1 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:18:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster2 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=1 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster2 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster2 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=1 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster2:2 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2:2 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-18T01:18:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster2:2 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster2 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=2 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster2 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster2 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=2 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster2:3 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2:3 2024-09-18T01:18:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster2:3 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster2 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=3 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster2 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster2 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=3 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:18:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ tee -a /tmp/tmp.YxZgdY10x3/rbd-mirror.test.log 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ date '+%F %T' 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo 2024-09-18 01:18:24 TEST: add image and test replay after client crashes 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:2024-09-18 01:18:24 TEST: add image and test replay after client crashes 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=test 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:18:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror test 512M 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:18:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:18:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-18T01:18:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:18:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10ad7d37fa82 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10ad7d37fa82 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:18:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:24 2024 2024-09-18T01:18:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:18:24 2024 2024-09-18T01:18:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:24 2024 2024-09-18T01:18:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:18:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:18:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:18:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:18:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:18:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:18:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:18:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:18:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:18:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:18:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:18:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:18:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:18:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:18:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:18:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:18:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:18:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:18:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:18:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:18:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:18:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:18:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:18:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:18:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:18:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:18:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:18:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:18:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:18:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:18:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:18:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:18:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:18:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:18:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:18:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:18:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:18:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:18:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:18:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-18T01:18:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:18:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:18:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:18:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:18:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-18T01:18:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:18:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 1 2024-09-18T01:18:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:18:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:18:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:18:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:18:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:18:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:18:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:18:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:18:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:18:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:18:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:18:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:18:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:18:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:18:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:18:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:18:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:18:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:18:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:18:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-18T01:18:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:18:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:18:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:18:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:18:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:18:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:18:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-18T01:18:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:18:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 2 2024-09-18T01:18:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:18:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:18:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:18:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:18:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:18:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:18:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:18:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:18:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:18:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:18:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:18:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:18:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:18:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-18T01:18:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:18:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:18:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:18:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:18:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-18T01:18:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:18:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 4 2024-09-18T01:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:18:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:18:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:18:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:18:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:18:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:18:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:18:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:18:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:18:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:18:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:18:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:18:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:18:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:18:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:18:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:18:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:18:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:18:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-18T01:18:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:18:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:18:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:18:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:18:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-18T01:18:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:18:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 8 2024-09-18T01:18:41.692 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6485046216121151) > weight (0.25) 2024-09-18T01:18:41.692 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-18T01:18:41.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-18T01:18:41.693 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-18T01:18:41.693 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-18T01:18:41.693 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:18:41.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:18:41.694+0000 7f9990de9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 36846) UID: 1000 2024-09-18T01:18:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:18:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:18:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:18:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:18:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:18:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:18:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:18:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:18:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:18:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:18:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:18:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:18:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:18:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:18:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:18:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:18:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:18:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:18:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-18T01:18:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:18:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:18:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:18:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:18:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-18T01:18:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:18:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 8 2024-09-18T01:18:47.795 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-18T01:18:47.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-18T01:18:47.796 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-18T01:18:47.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-18T01:18:47.799 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-18T01:18:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:18:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:18:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:18:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:18:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:18:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:18:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:18:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:18:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:18:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:18:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:18:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:18:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:18:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:18:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:18:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:18:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:18:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:18:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:18:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:18:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:18:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:18:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:18:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:18:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:18:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:18:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:18:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:18:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:18:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:18:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:18:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:18:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:18:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:18:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:18:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:18:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:18:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:18:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:18:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:18:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:18:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:18:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:18:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:18:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:18:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:18:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:18:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:18:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:18:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:18:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:18:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:18:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:18:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:18:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:18:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:18:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name= 2024-09-18T01:18:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 10 2024-09-18T01:18:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:18:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:18:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:18:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:18:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:18:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:18:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:18:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:18:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:18:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 49449984 bytes 2024-09-18T01:18:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 75904000 bytes 2024-09-18T01:18:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 99896320 bytes 2024-09-18T01:18:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 111312896 bytes 2024-09-18T01:18:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 201741312 bytes 2024-09-18T01:18:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 259452928 bytes 2024-09-18T01:18:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 283201536 bytes 2024-09-18T01:18:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 354147328 bytes 2024-09-18T01:18:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 367836160 bytes 2024-09-18T01:18:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 401534976 bytes 2024-09-18T01:18:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 472431616 bytes 2024-09-18T01:18:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 493714432 bytes 2024-09-18T01:18:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 637349888 bytes 2024-09-18T01:18:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 680863744 bytes 2024-09-18T01:18:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 742134784 bytes 2024-09-18T01:18:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 783997952 bytes 2024-09-18T01:18:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 801316864 bytes 2024-09-18T01:18:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 838960128 bytes 2024-09-18T01:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 850247680 bytes 2024-09-18T01:18:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 906491904 bytes 2024-09-18T01:18:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 948717568 bytes 2024-09-18T01:18:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1006401536 bytes 2024-09-18T01:18:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1026859008 bytes 2024-09-18T01:18:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1119081472 bytes 2024-09-18T01:18:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1119712256 bytes 2024-09-18T01:18:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1145232384 bytes 2024-09-18T01:18:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1192137728 bytes 2024-09-18T01:18:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1247417344 bytes 2024-09-18T01:18:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1314369536 bytes 2024-09-18T01:18:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1363557376 bytes 2024-09-18T01:18:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1475361792 bytes 2024-09-18T01:18:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1508385792 bytes 2024-09-18T01:18:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1573890048 bytes 2024-09-18T01:18:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1609786368 bytes 2024-09-18T01:18:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1656602624 bytes 2024-09-18T01:18:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1697156096 bytes 2024-09-18T01:18:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1745223680 bytes 2024-09-18T01:18:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1787446272 bytes 2024-09-18T01:18:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1833392128 bytes 2024-09-18T01:18:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1917562880 bytes 2024-09-18T01:18:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1924919296 bytes 2024-09-18T01:18:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1971207168 bytes 2024-09-18T01:18:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2039333888 bytes 2024-09-18T01:18:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2096600064 bytes 2024-09-18T01:18:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2110368768 bytes 2024-09-18T01:18:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2143346688 bytes 2024-09-18T01:18:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2243265536 bytes 2024-09-18T01:18:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2244418560 bytes 2024-09-18T01:18:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2302538752 bytes 2024-09-18T01:18:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2345370624 bytes 2024-09-18T01:18:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2377958400 bytes 2024-09-18T01:18:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2466138112 bytes 2024-09-18T01:18:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2494786560 bytes 2024-09-18T01:18:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2539729920 bytes 2024-09-18T01:18:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2576222208 bytes 2024-09-18T01:18:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2662891520 bytes 2024-09-18T01:18:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2704025600 bytes 2024-09-18T01:18:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2735464448 bytes 2024-09-18T01:18:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2781844480 bytes 2024-09-18T01:18:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2835017728 bytes 2024-09-18T01:18:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2860037120 bytes 2024-09-18T01:18:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2899018752 bytes 2024-09-18T01:18:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 3031730176 bytes 2024-09-18T01:18:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 3032327168 bytes 2024-09-18T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 3083644928 bytes 2024-09-18T01:18:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 3123396608 bytes 2024-09-18T01:18:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3174050816 bytes 2024-09-18T01:18:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3216776192 bytes 2024-09-18T01:18:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3243335680 bytes 2024-09-18T01:18:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3274277888 bytes 2024-09-18T01:18:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3354349568 bytes 2024-09-18T01:18:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3435307008 bytes 2024-09-18T01:18:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3511905280 bytes 2024-09-18T01:18:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3571150848 bytes 2024-09-18T01:18:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3643079680 bytes 2024-09-18T01:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3657693184 bytes 2024-09-18T01:18:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3713174528 bytes 2024-09-18T01:18:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3790842880 bytes 2024-09-18T01:18:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3836593152 bytes 2024-09-18T01:18:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3891474432 bytes 2024-09-18T01:18:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3926073344 bytes 2024-09-18T01:18:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3945266176 bytes 2024-09-18T01:18:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3961889792 bytes 2024-09-18T01:18:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 4025298944 bytes 2024-09-18T01:18:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 4072266752 bytes 2024-09-18T01:18:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 4093492224 bytes 2024-09-18T01:18:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 4105827328 bytes 2024-09-18T01:18:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 4131252224 bytes 2024-09-18T01:18:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 4151745536 bytes 2024-09-18T01:19:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 4189473792 bytes 2024-09-18T01:19:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4242203648 bytes 2024-09-18T01:19:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4298769408 bytes 2024-09-18T01:19:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4336474112 bytes 2024-09-18T01:19:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4357400576 bytes 2024-09-18T01:19:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4405509120 bytes 2024-09-18T01:19:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4518812672 bytes 2024-09-18T01:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4530432000 bytes 2024-09-18T01:19:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4604460032 bytes 2024-09-18T01:19:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:19:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:19:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:19:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:19:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:19:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:19:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:19:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:19:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:19:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:19:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:19:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: down+unknown 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: status not found 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: down+unknown 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: status not found 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: ' 2024-09-18T01:19:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:19:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 1 2024-09-18T01:19:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:19:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 2 2024-09-18T01:19:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:19:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:19:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:19:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:19:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:19:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:19:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:19:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: down+unknown 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: status not found 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: down+unknown 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: status not found 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: ' 2024-09-18T01:19:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:19:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 1 2024-09-18T01:19:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:19:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 4 2024-09-18T01:19:05.691 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8212658290574674) > weight (0.25) 2024-09-18T01:19:05.691 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9966324841293104) > weight (0.25) 2024-09-18T01:19:05.691 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6328374331446731) > weight (0.25) 2024-09-18T01:19:05.692 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2579341184701913) > weight (0.25) 2024-09-18T01:19:05.692 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-09-18T01:19:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:19:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: down+unknown 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: status not found 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: down+unknown 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: status not found 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: ' 2024-09-18T01:19:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:19:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 1 2024-09-18T01:19:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:19:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 8 2024-09-18T01:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:19:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:19:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:19:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:19:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622305,"remote_snapshot_timestamp":1726622305,"replay_state":"idle"} 2024-09-18T01:19:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:19:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:19:11 2024-09-18T01:19:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:19:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:19:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:19:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:19:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:19:11' 2024-09-18T01:19:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:19:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:19:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:19:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622305,"remote_snapshot_timestamp":1726622305,"replay_state":"idle"} 2024-09-18T01:19:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:19:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:19:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:19:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:19:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:19:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:19:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:19:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:19:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:19:11' 2024-09-18T01:19:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:19:11' 2024-09-18T01:19:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:19:11' 2024-09-18T01:19:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:19:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:19:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:19:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142104282mirror.0smithi11519.1.1-330-gc2ddf972falseOK42144283mirror.1smithi11519.1.1-330-gc2ddf972falseOK42184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622305,"remote_snapshot_timestamp":1726622305,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:19:11cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:19:11' 2024-09-18T01:19:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:19:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:19:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:19:11' 2024-09-18T01:19:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:19:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:19:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:19:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:19:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap1 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap1 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap1 2024-09-18T01:19:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:19:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:19:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:19:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:19:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:19:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:19:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:19:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:19:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:19:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:19:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:19:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:19:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:19:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:19:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:19:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:19:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:19:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:19:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:19:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:19:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:19:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:19:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:19:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:19:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:19:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:19:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:19:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:19:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:19:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:19:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:19:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:19:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:19:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:19:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:19:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:19:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:19:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:19:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:19:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:19:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:19:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:19:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:19:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:19:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:19:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:19:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:19:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:19:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:19:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:19:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:19:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:19:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:19:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:19:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:19:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:19:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:19:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:19:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:19:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:19:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:19:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:19:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:19:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:19:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:19:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:19:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:19:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:19:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:19:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:19:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:19:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:19:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:19:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:19:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:19:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:19:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:19:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:19:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 5 2024-09-18T01:19:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:19:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:19:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:19:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:19:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:19:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:19:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=5 2024-09-18T01:19:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:19:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:19:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:19:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:19:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:19:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:19:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:19:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:19:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:19:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:19:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:19:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 5 = '' 2024-09-18T01:19:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:19:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:19:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:19:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:19:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:19:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:19:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:19:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:19:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:19:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:19:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 5 = '' 2024-09-18T01:19:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:19:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:19:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:19:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:19:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:19:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:19:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:19:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:19:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:19:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:19:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 5 = '' 2024-09-18T01:19:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:19:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:19:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:19:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:19:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:19:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:19:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=5 2024-09-18T01:19:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 5 = 5 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap1 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:19:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:19:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-18T01:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap1 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap1 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:19:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:19:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:19:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:19:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:19:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:19:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:19:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:19:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:19:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap1-clean 2024-09-18T01:19:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-18T01:19:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:19:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap1-clean 2024-09-18T01:19:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:19:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:19:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:19:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:19:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:19:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:19:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:19:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:19:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 64663552 bytes 2024-09-18T01:19:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 108958720 bytes 2024-09-18T01:19:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 122448896 bytes 2024-09-18T01:19:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 158302208 bytes 2024-09-18T01:19:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 200867840 bytes 2024-09-18T01:19:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 286846976 bytes 2024-09-18T01:19:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 307147776 bytes 2024-09-18T01:19:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 369131520 bytes 2024-09-18T01:19:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 473751552 bytes 2024-09-18T01:19:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 529089536 bytes 2024-09-18T01:19:42.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 594430976 bytes 2024-09-18T01:19:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 620366848 bytes 2024-09-18T01:19:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 665172992 bytes 2024-09-18T01:19:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 751685632 bytes 2024-09-18T01:19:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 770618368 bytes 2024-09-18T01:19:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 811189248 bytes 2024-09-18T01:19:42.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 841425920 bytes 2024-09-18T01:19:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 903562240 bytes 2024-09-18T01:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 990345216 bytes 2024-09-18T01:19:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1033023488 bytes 2024-09-18T01:19:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1086801920 bytes 2024-09-18T01:19:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1118662656 bytes 2024-09-18T01:19:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1208492032 bytes 2024-09-18T01:19:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1217727488 bytes 2024-09-18T01:19:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1250259968 bytes 2024-09-18T01:19:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1290110976 bytes 2024-09-18T01:19:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1347318784 bytes 2024-09-18T01:19:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1388222464 bytes 2024-09-18T01:19:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1405186048 bytes 2024-09-18T01:19:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1406023680 bytes 2024-09-18T01:19:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1429704704 bytes 2024-09-18T01:19:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1503864832 bytes 2024-09-18T01:19:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1515779072 bytes 2024-09-18T01:19:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1572060160 bytes 2024-09-18T01:19:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1596388352 bytes 2024-09-18T01:19:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1626704896 bytes 2024-09-18T01:19:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1648121856 bytes 2024-09-18T01:19:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1679152128 bytes 2024-09-18T01:19:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1751672832 bytes 2024-09-18T01:19:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1797030912 bytes 2024-09-18T01:19:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1865144320 bytes 2024-09-18T01:19:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1899400192 bytes 2024-09-18T01:19:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1942756352 bytes 2024-09-18T01:19:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2016405504 bytes 2024-09-18T01:19:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2103580672 bytes 2024-09-18T01:19:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2104228864 bytes 2024-09-18T01:19:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2226751488 bytes 2024-09-18T01:19:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2245864448 bytes 2024-09-18T01:19:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2305742848 bytes 2024-09-18T01:19:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2332381184 bytes 2024-09-18T01:19:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2350112768 bytes 2024-09-18T01:19:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2350738432 bytes 2024-09-18T01:19:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2406080512 bytes 2024-09-18T01:19:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2481655808 bytes 2024-09-18T01:19:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2551966720 bytes 2024-09-18T01:19:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2591299584 bytes 2024-09-18T01:19:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2591912960 bytes 2024-09-18T01:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2604341248 bytes 2024-09-18T01:19:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2623759360 bytes 2024-09-18T01:19:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2664365056 bytes 2024-09-18T01:19:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2699814912 bytes 2024-09-18T01:19:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2714947584 bytes 2024-09-18T01:19:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2756419584 bytes 2024-09-18T01:19:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2794160128 bytes 2024-09-18T01:19:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2856497152 bytes 2024-09-18T01:19:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2939039744 bytes 2024-09-18T01:19:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2993961984 bytes 2024-09-18T01:19:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3029748736 bytes 2024-09-18T01:19:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3120787456 bytes 2024-09-18T01:19:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3201976320 bytes 2024-09-18T01:19:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3267672064 bytes 2024-09-18T01:19:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3303999488 bytes 2024-09-18T01:19:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3326867456 bytes 2024-09-18T01:19:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3390385152 bytes 2024-09-18T01:19:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3404100608 bytes 2024-09-18T01:19:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3404713984 bytes 2024-09-18T01:19:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3434342400 bytes 2024-09-18T01:19:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3457819648 bytes 2024-09-18T01:19:48.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3470547968 bytes 2024-09-18T01:19:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3500551168 bytes 2024-09-18T01:19:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3532790784 bytes 2024-09-18T01:19:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3603646464 bytes 2024-09-18T01:19:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3611316224 bytes 2024-09-18T01:19:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3671293952 bytes 2024-09-18T01:19:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3712189440 bytes 2024-09-18T01:19:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3794688000 bytes 2024-09-18T01:19:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3858828288 bytes 2024-09-18T01:19:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3863618560 bytes 2024-09-18T01:19:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3894112256 bytes 2024-09-18T01:19:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3894722560 bytes 2024-09-18T01:19:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3995454464 bytes 2024-09-18T01:19:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4010706944 bytes 2024-09-18T01:19:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4034750464 bytes 2024-09-18T01:19:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:19:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:19:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:19:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:19:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:19:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:19:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:19:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:19:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:19:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:19:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:19:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:19:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:19:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:19:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":17812206.93,"bytes_per_snapshot":267183104.0,"last_snapshot_bytes":534366208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622374,"remote_snapshot_timestamp":1726622374,"replay_state":"idle"} 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:19:41 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:19:41' 2024-09-18T01:19:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:19:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:19:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:19:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":17812206.93,"bytes_per_snapshot":267183104.0,"last_snapshot_bytes":534366208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622374,"remote_snapshot_timestamp":1726622374,"replay_state":"idle"} 2024-09-18T01:19:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:19:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:19:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:19:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:19:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:19:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:19:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:19:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:19:41' 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:19:41' 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:19:41' 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:19:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:19:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142104282mirror.0smithi11519.1.1-330-gc2ddf972falseOK42144283mirror.1smithi11519.1.1-330-gc2ddf972falseOK42184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":17812206.93,"bytes_per_snapshot":267183104.0,"last_snapshot_bytes":534366208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622374,"remote_snapshot_timestamp":1726622374,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:19:41cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:19:41' 2024-09-18T01:19:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:19:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:19:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:19:41' 2024-09-18T01:19:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:19:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:19:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:19:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:19:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:19:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:19:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap2 2024-09-18T01:19:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap2 2024-09-18T01:19:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:19:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:19:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:19:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap2 2024-09-18T01:19:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:20:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:20:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:20:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:20:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:20:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:20:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:20:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:20:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:20:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:20:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:20:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:20:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:20:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:20:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:20:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:20:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:20:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:20:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:20:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:20:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:20:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:20:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:20:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:20:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:20:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:20:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:20:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:20:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:20:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:20:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:20:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:20:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:20:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:20:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:20:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:20:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:20:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:20:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:20:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:20:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:20:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:20:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:20:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:20:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:20:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:20:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:20:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:20:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:20:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:20:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:20:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:20:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:20:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:20:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:20:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:20:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:20:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:20:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:20:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:20:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:20:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:20:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:20:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:20:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 8 2024-09-18T01:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:20:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:20:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:20:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=8 2024-09-18T01:20:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:20:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:20:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:20:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:20:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:20:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:20:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:20:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:20:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:20:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:20:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:20:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 8 = '' 2024-09-18T01:20:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:20:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:20:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:20:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:20:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:20:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:20:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:20:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:20:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:20:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:20:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 8 = '' 2024-09-18T01:20:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:20:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:20:24.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6694700662198) > weight (0.25) 2024-09-18T01:20:24.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8572353894020966) > weight (0.25) 2024-09-18T01:20:24.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5057007400031547) > weight (0.25) 2024-09-18T01:20:24.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8404046034824769) > weight (0.25) 2024-09-18T01:20:24.800 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-18T01:20:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:20:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 8 = '' 2024-09-18T01:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:20:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:20:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:20:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:20:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=8 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 8 = 8 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap2 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:20:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:20:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-18T01:20:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:20:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:20:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:20:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 4 2024-09-18T01:20:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:20:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:20:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:20:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:20:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:20:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:20:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap1-clean ']' 2024-09-18T01:20:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-18T01:20:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap1-clean 2024-09-18T01:20:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:20:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:20:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:20:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:20:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:20:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap2 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap2 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:20:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:20:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:20:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:20:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:20:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap2-clean 2024-09-18T01:20:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-18T01:20:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:20:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap2-clean 2024-09-18T01:20:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:20:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:20:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:20:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:20:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:20:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:20:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:20:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:20:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 46340096 bytes 2024-09-18T01:20:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 95982592 bytes 2024-09-18T01:20:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 147297280 bytes 2024-09-18T01:20:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 197454848 bytes 2024-09-18T01:20:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 252493824 bytes 2024-09-18T01:20:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 268598272 bytes 2024-09-18T01:20:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 321125376 bytes 2024-09-18T01:20:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 325410816 bytes 2024-09-18T01:20:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 429548544 bytes 2024-09-18T01:20:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 484349952 bytes 2024-09-18T01:20:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 544178176 bytes 2024-09-18T01:20:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 604086272 bytes 2024-09-18T01:20:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 649865216 bytes 2024-09-18T01:20:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 670175232 bytes 2024-09-18T01:20:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 769398784 bytes 2024-09-18T01:20:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 840759296 bytes 2024-09-18T01:20:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 884477952 bytes 2024-09-18T01:20:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 927960064 bytes 2024-09-18T01:20:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 991456256 bytes 2024-09-18T01:20:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1003294720 bytes 2024-09-18T01:20:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1069522944 bytes 2024-09-18T01:20:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1145688064 bytes 2024-09-18T01:20:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1175984128 bytes 2024-09-18T01:20:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1176602624 bytes 2024-09-18T01:20:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1203015680 bytes 2024-09-18T01:20:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1275057152 bytes 2024-09-18T01:20:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1306570752 bytes 2024-09-18T01:20:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1343255552 bytes 2024-09-18T01:20:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1389964288 bytes 2024-09-18T01:20:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1393179648 bytes 2024-09-18T01:20:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1435783168 bytes 2024-09-18T01:20:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1498970112 bytes 2024-09-18T01:20:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1550779392 bytes 2024-09-18T01:20:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1568062464 bytes 2024-09-18T01:20:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1610520576 bytes 2024-09-18T01:20:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1632586752 bytes 2024-09-18T01:20:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1669006336 bytes 2024-09-18T01:20:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1712982016 bytes 2024-09-18T01:20:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1730824192 bytes 2024-09-18T01:20:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1797248000 bytes 2024-09-18T01:20:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1873636352 bytes 2024-09-18T01:20:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1894553600 bytes 2024-09-18T01:20:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1914430464 bytes 2024-09-18T01:20:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1950359552 bytes 2024-09-18T01:20:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2116257792 bytes 2024-09-18T01:20:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2186509312 bytes 2024-09-18T01:20:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2236378112 bytes 2024-09-18T01:20:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2262719488 bytes 2024-09-18T01:20:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2287890432 bytes 2024-09-18T01:20:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2314793984 bytes 2024-09-18T01:20:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2325823488 bytes 2024-09-18T01:20:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2332838912 bytes 2024-09-18T01:20:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2347826176 bytes 2024-09-18T01:20:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2381956096 bytes 2024-09-18T01:20:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2382587904 bytes 2024-09-18T01:20:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2446093312 bytes 2024-09-18T01:20:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2501648384 bytes 2024-09-18T01:20:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2544013312 bytes 2024-09-18T01:20:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2587955200 bytes 2024-09-18T01:20:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2627296256 bytes 2024-09-18T01:20:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2678369280 bytes 2024-09-18T01:20:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2731326464 bytes 2024-09-18T01:20:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2768488448 bytes 2024-09-18T01:20:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2810717184 bytes 2024-09-18T01:20:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2830061568 bytes 2024-09-18T01:20:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2945527808 bytes 2024-09-18T01:20:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3031088128 bytes 2024-09-18T01:20:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3065860096 bytes 2024-09-18T01:20:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3103259648 bytes 2024-09-18T01:20:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3158839296 bytes 2024-09-18T01:20:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3198930944 bytes 2024-09-18T01:20:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3302737920 bytes 2024-09-18T01:20:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3357454336 bytes 2024-09-18T01:20:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3379842048 bytes 2024-09-18T01:20:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3410554880 bytes 2024-09-18T01:20:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3411165184 bytes 2024-09-18T01:20:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3509074944 bytes 2024-09-18T01:20:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3568869376 bytes 2024-09-18T01:20:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3622206464 bytes 2024-09-18T01:20:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3663490048 bytes 2024-09-18T01:20:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3752907776 bytes 2024-09-18T01:20:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3798251520 bytes 2024-09-18T01:20:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3825985536 bytes 2024-09-18T01:20:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3892940800 bytes 2024-09-18T01:20:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3926896640 bytes 2024-09-18T01:20:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3970634752 bytes 2024-09-18T01:20:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3971233792 bytes 2024-09-18T01:20:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3977670656 bytes 2024-09-18T01:20:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 4035761152 bytes 2024-09-18T01:20:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 4074080256 bytes 2024-09-18T01:20:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4159416320 bytes 2024-09-18T01:20:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4225343488 bytes 2024-09-18T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:20:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:20:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:20:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:20:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:20:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:20:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:20:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:20:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:20:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":17812206.93,"bytes_per_snapshot":267183104.0,"last_snapshot_bytes":534366208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622374,"remote_snapshot_timestamp":1726622374,"replay_state":"idle"} 2024-09-18T01:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:20:11 2024-09-18T01:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:20:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:20:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:20:41' 2024-09-18T01:20:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:20:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:20:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:20:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":17812206.93,"bytes_per_snapshot":267183104.0,"last_snapshot_bytes":534366208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622374,"remote_snapshot_timestamp":1726622374,"replay_state":"idle"} 2024-09-18T01:20:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:20:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:20:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:20:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:20:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:20:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:20:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:20:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:20:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:20:11' 2024-09-18T01:20:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:20:11' 2024-09-18T01:20:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:20:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:20:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:20:11' 2024-09-18T01:20:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:20:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:20:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:20:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142104282mirror.0smithi11519.1.1-330-gc2ddf972falseOK42144283mirror.1smithi11519.1.1-330-gc2ddf972falseOK42184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":17812206.93,"bytes_per_snapshot":267183104.0,"last_snapshot_bytes":534366208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622374,"remote_snapshot_timestamp":1726622374,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:20:11cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:20:41' 2024-09-18T01:20:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:20:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:20:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:20:11' 2024-09-18T01:20:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:20:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:20:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:20:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:20:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:20:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:20:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:20:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap3 2024-09-18T01:20:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap3 2024-09-18T01:20:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:20:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:20:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:20:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap3 2024-09-18T01:20:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:20:57.692 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8109762664903837) > weight (0.25) 2024-09-18T01:20:57.693 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.40127342928436094) > weight (0.25) 2024-09-18T01:20:57.693 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8886795947643152) > weight (0.25) 2024-09-18T01:20:57.693 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6078255441063762) > weight (0.25) 2024-09-18T01:20:57.693 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-09-18T01:21:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:21:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:21:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:21:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:21:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:21:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:21:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:21:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:21:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:21:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:21:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:21:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:21:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:21:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:21:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:21:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:21:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:21:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:21:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:21:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:21:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:21:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:21:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:21:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:21:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:21:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:21:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:21:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:21:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:21:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:21:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:21:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:21:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:21:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:21:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:21:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:21:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:21:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:21:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:21:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:21:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:21:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:21:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:21:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:21:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:21:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:21:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:21:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:21:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:21:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:21:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:21:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:21:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:21:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 11 2024-09-18T01:21:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:21:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:21:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:21:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:21:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:21:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:21:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=11 2024-09-18T01:21:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:21:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:21:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:21:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:21:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:21:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:21:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:21:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:21:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:21:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:21:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:21:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 11 = '' 2024-09-18T01:21:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:21:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:21:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:21:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:21:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:21:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:21:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:21:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:21:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 11 = '' 2024-09-18T01:21:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:21:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:21:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:21:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:21:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:21:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:21:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:21:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:21:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:21:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:21:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 11 = '' 2024-09-18T01:21:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:21:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:21:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:21:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:21:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:21:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:21:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=11 2024-09-18T01:21:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 11 = 11 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap3 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:21:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:21:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-18T01:21:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:21:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:21:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:21:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 6 2024-09-18T01:21:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:21:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:21:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:20:28 2024 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:21:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:21:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:21:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap2-clean ']' 2024-09-18T01:21:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-18T01:21:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap2-clean 2024-09-18T01:21:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:21:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:21:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:21:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:21:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:21:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap3 2024-09-18T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap3 2024-09-18T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:21:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:21:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:21:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:21:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:21:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap3-clean 2024-09-18T01:21:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-18T01:21:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:21:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap3-clean 2024-09-18T01:21:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:21:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:21:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:21:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:21:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:21:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=11 2024-09-18T01:21:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:21:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:21:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 72125440 bytes 2024-09-18T01:21:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 125018112 bytes 2024-09-18T01:21:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 127590400 bytes 2024-09-18T01:21:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 155595776 bytes 2024-09-18T01:21:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 212795392 bytes 2024-09-18T01:21:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 273870848 bytes 2024-09-18T01:21:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 344631296 bytes 2024-09-18T01:21:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 365941760 bytes 2024-09-18T01:21:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 440612864 bytes 2024-09-18T01:21:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 466734080 bytes 2024-09-18T01:21:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 510186496 bytes 2024-09-18T01:21:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 569611264 bytes 2024-09-18T01:21:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 599089152 bytes 2024-09-18T01:21:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 616750080 bytes 2024-09-18T01:21:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 674153472 bytes 2024-09-18T01:21:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 675730432 bytes 2024-09-18T01:21:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 689968128 bytes 2024-09-18T01:21:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 748798976 bytes 2024-09-18T01:21:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 793375744 bytes 2024-09-18T01:21:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 848797696 bytes 2024-09-18T01:21:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 862198784 bytes 2024-09-18T01:21:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 882288640 bytes 2024-09-18T01:21:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 931060736 bytes 2024-09-18T01:21:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 964745216 bytes 2024-09-18T01:21:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 969247744 bytes 2024-09-18T01:21:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1003554816 bytes 2024-09-18T01:21:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1110571008 bytes 2024-09-18T01:21:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1186618368 bytes 2024-09-18T01:21:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1217000448 bytes 2024-09-18T01:21:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1301427200 bytes 2024-09-18T01:21:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1328751616 bytes 2024-09-18T01:21:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1387653120 bytes 2024-09-18T01:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1403555840 bytes 2024-09-18T01:21:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1473057792 bytes 2024-09-18T01:21:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1487958016 bytes 2024-09-18T01:21:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1573635072 bytes 2024-09-18T01:21:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1636439040 bytes 2024-09-18T01:21:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1704923136 bytes 2024-09-18T01:21:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1789475840 bytes 2024-09-18T01:21:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1856994304 bytes 2024-09-18T01:21:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1884332032 bytes 2024-09-18T01:21:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1986709504 bytes 2024-09-18T01:21:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2070774784 bytes 2024-09-18T01:21:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2125253632 bytes 2024-09-18T01:21:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2157173760 bytes 2024-09-18T01:21:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2185175040 bytes 2024-09-18T01:21:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2197224448 bytes 2024-09-18T01:21:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2222569472 bytes 2024-09-18T01:21:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2283556864 bytes 2024-09-18T01:21:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2340411392 bytes 2024-09-18T01:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2385299456 bytes 2024-09-18T01:21:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2446917632 bytes 2024-09-18T01:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2508782592 bytes 2024-09-18T01:21:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2550152192 bytes 2024-09-18T01:21:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2644775936 bytes 2024-09-18T01:21:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2659640320 bytes 2024-09-18T01:21:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2722280448 bytes 2024-09-18T01:21:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2800304128 bytes 2024-09-18T01:21:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2844389376 bytes 2024-09-18T01:21:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2854189056 bytes 2024-09-18T01:21:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2895811584 bytes 2024-09-18T01:21:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2916257792 bytes 2024-09-18T01:21:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2986398720 bytes 2024-09-18T01:21:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 3040867328 bytes 2024-09-18T01:21:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 3092451328 bytes 2024-09-18T01:21:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 3134930944 bytes 2024-09-18T01:21:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3231608832 bytes 2024-09-18T01:21:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3261546496 bytes 2024-09-18T01:21:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3262152704 bytes 2024-09-18T01:21:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3323199488 bytes 2024-09-18T01:21:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3384582144 bytes 2024-09-18T01:21:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3431750656 bytes 2024-09-18T01:21:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3466008576 bytes 2024-09-18T01:21:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3563815936 bytes 2024-09-18T01:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3627807744 bytes 2024-09-18T01:21:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3654708224 bytes 2024-09-18T01:21:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3763646464 bytes 2024-09-18T01:21:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3844066304 bytes 2024-09-18T01:21:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3950510080 bytes 2024-09-18T01:21:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3995594752 bytes 2024-09-18T01:21:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 4059271168 bytes 2024-09-18T01:21:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 4125025280 bytes 2024-09-18T01:21:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 4180408320 bytes 2024-09-18T01:21:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 4221758464 bytes 2024-09-18T01:21:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 4267716608 bytes 2024-09-18T01:21:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 4294278144 bytes 2024-09-18T01:21:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 4341361664 bytes 2024-09-18T01:21:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 4356449280 bytes 2024-09-18T01:21:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 4379742208 bytes 2024-09-18T01:21:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 4405161984 bytes 2024-09-18T01:21:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4431412224 bytes 2024-09-18T01:21:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4529555456 bytes 2024-09-18T01:21:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4580727808 bytes 2024-09-18T01:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4607036416 bytes 2024-09-18T01:21:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4616369152 bytes 2024-09-18T01:21:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4616997888 bytes 2024-09-18T01:21:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4665798656 bytes 2024-09-18T01:21:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4710451200 bytes 2024-09-18T01:21:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4762082304 bytes 2024-09-18T01:21:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4798410752 bytes 2024-09-18T01:21:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4864382976 bytes 2024-09-18T01:21:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4907592704 bytes 2024-09-18T01:21:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 5004402688 bytes 2024-09-18T01:21:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 5064107008 bytes 2024-09-18T01:21:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 5093362688 bytes 2024-09-18T01:21:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 5181711360 bytes 2024-09-18T01:21:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 5223298048 bytes 2024-09-18T01:21:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 5266722816 bytes 2024-09-18T01:21:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 5304090624 bytes 2024-09-18T01:21:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11000 IOs, 5356662784 bytes 2024-09-18T01:21:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11100 IOs, 5389267968 bytes 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:21:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:21:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:21:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534680064.0,"last_snapshot_bytes":534993920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622422,"remote_snapshot_timestamp":1726622422,"replay_state":"idle"} 2024-09-18T01:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:21:11 2024-09-18T01:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:21:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:21:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:21:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:21:11' 2024-09-18T01:21:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:21:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:21:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:21:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534680064.0,"last_snapshot_bytes":534993920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622422,"remote_snapshot_timestamp":1726622422,"replay_state":"idle"} 2024-09-18T01:21:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:21:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:21:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:21:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:21:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:21:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:21:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:21:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:21:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:21:11' 2024-09-18T01:21:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:21:11' 2024-09-18T01:21:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:21:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:21:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:21:11' 2024-09-18T01:21:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:21:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:21:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:21:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142104282mirror.0smithi11519.1.1-330-gc2ddf972falseOK42144283mirror.1smithi11519.1.1-330-gc2ddf972falseOK42184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534680064.0,"last_snapshot_bytes":534993920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622422,"remote_snapshot_timestamp":1726622422,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:21:11cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:21:11' 2024-09-18T01:21:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:21:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:21:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:21:11' 2024-09-18T01:21:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:21:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:21:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:21:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:21:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap4 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap4 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap4 2024-09-18T01:21:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:21:50.801 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7737567018424414) > weight (0.25) 2024-09-18T01:21:50.801 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7044693670047647) > weight (0.25) 2024-09-18T01:21:50.801 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-18T01:21:50.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-18T01:21:50.802 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-18T01:21:50.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:21:50.803+0000 7f9c7faa2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 36863) UID: 1000 2024-09-18T01:21:52.803 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-18T01:21:52.803 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:21:52.803 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-18T01:21:52.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-18T01:21:52.804 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-18T01:21:52.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-18T01:21:52.807 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-18T01:22:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:22:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:22:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:22:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:22:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:22:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:22:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:22:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:22:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:22:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:22:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:22:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:22:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:22:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:22:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:22:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:22:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:22:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:22:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:22:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:22:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:22:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:22:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:22:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:22:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:22:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:22:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:22:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:22:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:22:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:22:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:22:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:22:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:22:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:22:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:22:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:22:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:22:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:22:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:22:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:22:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:22:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:22:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:22:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:22:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:22:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:22:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:22:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 14 2024-09-18T01:22:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:22:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:22:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:22:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:22:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:22:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:22:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=14 2024-09-18T01:22:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:22:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:22:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:22:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:22:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:22:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:22:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:22:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:22:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:22:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:22:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:22:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 14 = '' 2024-09-18T01:22:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:22:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:22:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:22:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:22:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:22:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:22:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:22:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:22:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:22:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:22:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 14 = '' 2024-09-18T01:22:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:22:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:22:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:22:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:22:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:22:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:22:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:22:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:22:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 14 = '' 2024-09-18T01:22:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:22:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:22:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:22:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:22:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:22:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:22:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=14 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 14 = 14 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap4 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:22:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:22:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-18T01:22:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:22:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:22:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 8 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:20:28 2024 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:22:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:22:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:22:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap3-clean ']' 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap3-clean 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:22:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:22:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:22:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:22:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap4 2024-09-18T01:22:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap4 2024-09-18T01:22:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:22:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:22:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:22:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:22:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:22:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:22:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:22:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap4-clean 2024-09-18T01:22:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-18T01:22:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:22:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap4-clean 2024-09-18T01:22:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:22:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=14 2024-09-18T01:22:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:22:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:22:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 51933184 bytes 2024-09-18T01:22:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 76311552 bytes 2024-09-18T01:22:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 106330112 bytes 2024-09-18T01:22:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 168941568 bytes 2024-09-18T01:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 224083968 bytes 2024-09-18T01:22:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 287196160 bytes 2024-09-18T01:22:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 289334272 bytes 2024-09-18T01:22:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 416622592 bytes 2024-09-18T01:22:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 441298944 bytes 2024-09-18T01:22:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 496386048 bytes 2024-09-18T01:22:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 506287104 bytes 2024-09-18T01:22:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 514776064 bytes 2024-09-18T01:22:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 561413120 bytes 2024-09-18T01:22:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 562054144 bytes 2024-09-18T01:22:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 605070336 bytes 2024-09-18T01:22:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 722958336 bytes 2024-09-18T01:22:16.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 767598592 bytes 2024-09-18T01:22:16.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 801862656 bytes 2024-09-18T01:22:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 846189568 bytes 2024-09-18T01:22:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 870303744 bytes 2024-09-18T01:22:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 917565440 bytes 2024-09-18T01:22:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 950595584 bytes 2024-09-18T01:22:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1000335360 bytes 2024-09-18T01:22:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1045470208 bytes 2024-09-18T01:22:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1119513600 bytes 2024-09-18T01:22:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1156685824 bytes 2024-09-18T01:22:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1182273536 bytes 2024-09-18T01:22:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1227804672 bytes 2024-09-18T01:22:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1252915200 bytes 2024-09-18T01:22:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1295895552 bytes 2024-09-18T01:22:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1321522176 bytes 2024-09-18T01:22:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1384871936 bytes 2024-09-18T01:22:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1404706816 bytes 2024-09-18T01:22:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1418387456 bytes 2024-09-18T01:22:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1482880000 bytes 2024-09-18T01:22:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1548524544 bytes 2024-09-18T01:22:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1611511808 bytes 2024-09-18T01:22:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1649767424 bytes 2024-09-18T01:22:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1696940032 bytes 2024-09-18T01:22:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1759011840 bytes 2024-09-18T01:22:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1802617856 bytes 2024-09-18T01:22:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1804517376 bytes 2024-09-18T01:22:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1837706240 bytes 2024-09-18T01:22:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1897466880 bytes 2024-09-18T01:22:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1912250368 bytes 2024-09-18T01:22:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1925667840 bytes 2024-09-18T01:22:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1961004032 bytes 2024-09-18T01:22:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1999469568 bytes 2024-09-18T01:22:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2053036032 bytes 2024-09-18T01:22:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2089363456 bytes 2024-09-18T01:22:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2145495040 bytes 2024-09-18T01:22:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2182192128 bytes 2024-09-18T01:22:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2270425088 bytes 2024-09-18T01:22:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2342343680 bytes 2024-09-18T01:22:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2419263488 bytes 2024-09-18T01:22:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2444213248 bytes 2024-09-18T01:22:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2498008064 bytes 2024-09-18T01:22:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2528793600 bytes 2024-09-18T01:22:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2564781056 bytes 2024-09-18T01:22:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2574745600 bytes 2024-09-18T01:22:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2592668672 bytes 2024-09-18T01:22:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2661801984 bytes 2024-09-18T01:22:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2694726656 bytes 2024-09-18T01:22:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2755672064 bytes 2024-09-18T01:22:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2766233600 bytes 2024-09-18T01:22:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2820335616 bytes 2024-09-18T01:22:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2841272320 bytes 2024-09-18T01:22:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2866665472 bytes 2024-09-18T01:22:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2922263552 bytes 2024-09-18T01:22:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2970381312 bytes 2024-09-18T01:22:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2981441536 bytes 2024-09-18T01:22:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3031280640 bytes 2024-09-18T01:22:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3102461952 bytes 2024-09-18T01:22:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3140897792 bytes 2024-09-18T01:22:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3153952768 bytes 2024-09-18T01:22:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3173920768 bytes 2024-09-18T01:22:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3216136192 bytes 2024-09-18T01:22:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3249573888 bytes 2024-09-18T01:22:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3327348736 bytes 2024-09-18T01:22:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3331145728 bytes 2024-09-18T01:22:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3369062400 bytes 2024-09-18T01:22:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3387089920 bytes 2024-09-18T01:22:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3421186048 bytes 2024-09-18T01:22:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3484564480 bytes 2024-09-18T01:22:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3555715072 bytes 2024-09-18T01:22:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3607005184 bytes 2024-09-18T01:22:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3656541184 bytes 2024-09-18T01:22:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3682580480 bytes 2024-09-18T01:22:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3751672832 bytes 2024-09-18T01:22:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3833615360 bytes 2024-09-18T01:22:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3867068416 bytes 2024-09-18T01:22:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3910994944 bytes 2024-09-18T01:22:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 3974016000 bytes 2024-09-18T01:22:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4037187584 bytes 2024-09-18T01:22:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4060530688 bytes 2024-09-18T01:22:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4175897600 bytes 2024-09-18T01:22:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4194304000 bytes 2024-09-18T01:22:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4245576704 bytes 2024-09-18T01:22:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4277097472 bytes 2024-09-18T01:22:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4344714240 bytes 2024-09-18T01:22:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4358203392 bytes 2024-09-18T01:22:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4395869184 bytes 2024-09-18T01:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4463931392 bytes 2024-09-18T01:22:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 4467222528 bytes 2024-09-18T01:22:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4525750272 bytes 2024-09-18T01:22:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 4593941504 bytes 2024-09-18T01:22:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 4671382528 bytes 2024-09-18T01:22:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 4700498944 bytes 2024-09-18T01:22:25.694 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-18T01:22:25.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-18T01:22:25.694 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-18T01:22:25.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:22:25.694+0000 7f70f3e8b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36692) UID: 1000 2024-09-18T01:22:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 4712906752 bytes 2024-09-18T01:22:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11000 IOs, 4778053632 bytes 2024-09-18T01:22:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11100 IOs, 4807154688 bytes 2024-09-18T01:22:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11200 IOs, 4836596736 bytes 2024-09-18T01:22:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11300 IOs, 4909012992 bytes 2024-09-18T01:22:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11400 IOs, 4909636608 bytes 2024-09-18T01:22:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11500 IOs, 4945037312 bytes 2024-09-18T01:22:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11600 IOs, 4954051584 bytes 2024-09-18T01:22:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11700 IOs, 5029695488 bytes 2024-09-18T01:22:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11800 IOs, 5094445056 bytes 2024-09-18T01:22:26.695 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-18T01:22:26.695 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:22:26.696 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-18T01:22:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-18T01:22:26.696 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-18T01:22:26.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-18T01:22:26.698 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-18T01:22:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11900 IOs, 5166150656 bytes 2024-09-18T01:22:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12000 IOs, 5237956608 bytes 2024-09-18T01:22:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12100 IOs, 5279557632 bytes 2024-09-18T01:22:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12200 IOs, 5319363584 bytes 2024-09-18T01:22:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12300 IOs, 5334293504 bytes 2024-09-18T01:22:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12400 IOs, 5383279616 bytes 2024-09-18T01:22:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12500 IOs, 5399075840 bytes 2024-09-18T01:22:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12600 IOs, 5463988224 bytes 2024-09-18T01:22:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12700 IOs, 5547069440 bytes 2024-09-18T01:22:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12800 IOs, 5637057536 bytes 2024-09-18T01:22:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12900 IOs, 5696089088 bytes 2024-09-18T01:22:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13000 IOs, 5698681856 bytes 2024-09-18T01:22:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13100 IOs, 5750254592 bytes 2024-09-18T01:22:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13200 IOs, 5790663680 bytes 2024-09-18T01:22:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13300 IOs, 5859589120 bytes 2024-09-18T01:22:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13400 IOs, 5920050176 bytes 2024-09-18T01:22:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13500 IOs, 5950009344 bytes 2024-09-18T01:22:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13600 IOs, 5975736320 bytes 2024-09-18T01:22:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13700 IOs, 5998405632 bytes 2024-09-18T01:22:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:13800 IOs, 6054443008 bytes 2024-09-18T01:22:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:22:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:22:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:22:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:22:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:22:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:22:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:22:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:22:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:22:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:22:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:22:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:22:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:22:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":3774873.6,"bytes_per_snapshot":536677888.0,"last_snapshot_bytes":536725504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622526,"remote_snapshot_timestamp":1726622526,"replay_state":"idle"} 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:22:11 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:22:11' 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:22:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:22:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:22:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":3774873.6,"bytes_per_snapshot":536677888.0,"last_snapshot_bytes":536725504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622526,"remote_snapshot_timestamp":1726622526,"replay_state":"idle"} 2024-09-18T01:22:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:22:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:22:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:22:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:22:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:22:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:22:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:22:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:22:11' 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:22:11' 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:22:11' 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:22:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='WARNINGWARNINGOK142144283mirror.1smithi11519.1.1-330-gc2ddf972falseOK42184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOK4431mirror.0smithi11519.1.1-330-gc2ddf972falseWARNINGnot reporting statustest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":3774873.6,"bytes_per_snapshot":536677888.0,"last_snapshot_bytes":536725504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622526,"remote_snapshot_timestamp":1726622526,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:22:11cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:22:11' 2024-09-18T01:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:22:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:22:11' 2024-09-18T01:22:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:22:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:22:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:22:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:22:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:22:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:22:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:22:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap5 2024-09-18T01:22:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap5 2024-09-18T01:22:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:22:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:22:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:22:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap5 2024-09-18T01:22:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:23:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:23:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:23:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:23:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:23:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:23:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:23:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:23:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:23:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:23:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:23:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:23:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:23:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:23:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:23:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:23:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:23:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:23:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:23:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:23:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:23:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:23:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:23:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:23:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:23:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:23:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:23:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:23:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:23:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:23:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:23:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:23:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:23:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:23:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:23:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:23:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:23:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:23:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:23:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:23:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:23:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:23:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:23:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:23:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:23:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:23:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:23:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:23:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:23:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:23:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:23:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:23:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:23:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:23:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:23:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:23:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:23:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:23:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:23:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:23:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:23:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:23:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:23:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:23:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:23:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:23:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:23:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:23:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:23:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 17 2024-09-18T01:23:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:23:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:23:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:23:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:23:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=17 2024-09-18T01:23:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:23:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:23:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:23:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:23:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:23:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:23:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:23:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:23:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:23:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:23:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 17 = '' 2024-09-18T01:23:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:23:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:23:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:23:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:23:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:23:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:23:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:23:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:23:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:23:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 17 = '' 2024-09-18T01:23:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:23:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:23:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:23:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:23:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:23:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:23:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:23:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:23:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:23:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 17 = '' 2024-09-18T01:23:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:23:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:23:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:23:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:23:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=17 2024-09-18T01:23:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 17 = 17 2024-09-18T01:23:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-18T01:23:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap5 2024-09-18T01:23:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:23:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:23:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:23:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-18T01:23:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 10 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:23:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:23:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:23:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:22:12 2024 2024-09-18T01:23:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:23:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:23:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:23:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:23:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:23:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:23:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap4-clean ']' 2024-09-18T01:23:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-18T01:23:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap4-clean 2024-09-18T01:23:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:23:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:23:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:23:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:23:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:23:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap5 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap5 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:23:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:23:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:23:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap5-clean 2024-09-18T01:23:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-18T01:23:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap5-clean 2024-09-18T01:23:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:23:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:23:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:23:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:23:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=10 2024-09-18T01:23:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:23:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:23:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 52807680 bytes 2024-09-18T01:23:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 63294464 bytes 2024-09-18T01:23:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 144309248 bytes 2024-09-18T01:23:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 192564224 bytes 2024-09-18T01:23:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 268043264 bytes 2024-09-18T01:23:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 291356672 bytes 2024-09-18T01:23:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 311321600 bytes 2024-09-18T01:23:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 330786816 bytes 2024-09-18T01:23:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 357532672 bytes 2024-09-18T01:23:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 386325504 bytes 2024-09-18T01:23:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 432501760 bytes 2024-09-18T01:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 488335360 bytes 2024-09-18T01:23:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 523555840 bytes 2024-09-18T01:23:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 575756288 bytes 2024-09-18T01:23:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 633593856 bytes 2024-09-18T01:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 692147200 bytes 2024-09-18T01:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 709770240 bytes 2024-09-18T01:23:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 755251200 bytes 2024-09-18T01:23:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 780672000 bytes 2024-09-18T01:23:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 808875008 bytes 2024-09-18T01:23:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 815426560 bytes 2024-09-18T01:23:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 870821888 bytes 2024-09-18T01:23:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 882581504 bytes 2024-09-18T01:23:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 930371584 bytes 2024-09-18T01:23:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 979362816 bytes 2024-09-18T01:23:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1031618560 bytes 2024-09-18T01:23:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1097181184 bytes 2024-09-18T01:23:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1157946368 bytes 2024-09-18T01:23:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1212335104 bytes 2024-09-18T01:23:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1233462272 bytes 2024-09-18T01:23:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1283767296 bytes 2024-09-18T01:23:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1322787840 bytes 2024-09-18T01:23:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1386089472 bytes 2024-09-18T01:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1443652608 bytes 2024-09-18T01:23:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1477983232 bytes 2024-09-18T01:23:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1519505408 bytes 2024-09-18T01:23:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1572220928 bytes 2024-09-18T01:23:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1630658560 bytes 2024-09-18T01:23:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1653263360 bytes 2024-09-18T01:23:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1667389440 bytes 2024-09-18T01:23:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1727564800 bytes 2024-09-18T01:23:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1817674752 bytes 2024-09-18T01:23:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1858557952 bytes 2024-09-18T01:23:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1912482816 bytes 2024-09-18T01:23:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1967244288 bytes 2024-09-18T01:23:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2028578816 bytes 2024-09-18T01:23:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2087042048 bytes 2024-09-18T01:23:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2136348672 bytes 2024-09-18T01:23:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2216097792 bytes 2024-09-18T01:23:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2236816384 bytes 2024-09-18T01:23:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2291965952 bytes 2024-09-18T01:23:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2320773120 bytes 2024-09-18T01:23:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2367304704 bytes 2024-09-18T01:23:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2396724224 bytes 2024-09-18T01:23:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2470910976 bytes 2024-09-18T01:23:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2501095424 bytes 2024-09-18T01:23:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2501706752 bytes 2024-09-18T01:23:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2520094720 bytes 2024-09-18T01:23:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2543352832 bytes 2024-09-18T01:23:17.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2552763392 bytes 2024-09-18T01:23:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2614424576 bytes 2024-09-18T01:23:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2667673600 bytes 2024-09-18T01:23:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2715976704 bytes 2024-09-18T01:23:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2725591040 bytes 2024-09-18T01:23:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2745034752 bytes 2024-09-18T01:23:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2770622464 bytes 2024-09-18T01:23:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2859979776 bytes 2024-09-18T01:23:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2903458816 bytes 2024-09-18T01:23:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2980524032 bytes 2024-09-18T01:23:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3040242688 bytes 2024-09-18T01:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3129059328 bytes 2024-09-18T01:23:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3156392960 bytes 2024-09-18T01:23:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3174964224 bytes 2024-09-18T01:23:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3274861568 bytes 2024-09-18T01:23:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3299914752 bytes 2024-09-18T01:23:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3321142272 bytes 2024-09-18T01:23:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3379335168 bytes 2024-09-18T01:23:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3395239936 bytes 2024-09-18T01:23:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3436200960 bytes 2024-09-18T01:23:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3465331712 bytes 2024-09-18T01:23:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3495110656 bytes 2024-09-18T01:23:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3542013952 bytes 2024-09-18T01:23:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3583184896 bytes 2024-09-18T01:23:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3641483264 bytes 2024-09-18T01:23:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3656639488 bytes 2024-09-18T01:23:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3658916864 bytes 2024-09-18T01:23:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3712782336 bytes 2024-09-18T01:23:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3767475200 bytes 2024-09-18T01:23:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3805859840 bytes 2024-09-18T01:23:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3928718336 bytes 2024-09-18T01:23:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3986851840 bytes 2024-09-18T01:23:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4006147072 bytes 2024-09-18T01:23:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4079197184 bytes 2024-09-18T01:23:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4147093504 bytes 2024-09-18T01:23:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4173811712 bytes 2024-09-18T01:23:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4194816000 bytes 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:23:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:23:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:23:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:23:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:23:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:23:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:23:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:23:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:23:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:23:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:23:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":17892352.0,"bytes_per_snapshot":536748032.0,"last_snapshot_bytes":536770560,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622582,"remote_snapshot_timestamp":1726622582,"replay_state":"idle"} 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:23:11 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:23:11' 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:23:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:23:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":17892352.0,"bytes_per_snapshot":536748032.0,"last_snapshot_bytes":536770560,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622582,"remote_snapshot_timestamp":1726622582,"replay_state":"idle"} 2024-09-18T01:23:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:23:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:23:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:23:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:23:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:23:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:23:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:23:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:23:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:23:11' 2024-09-18T01:23:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:23:11' 2024-09-18T01:23:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:23:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:23:11' 2024-09-18T01:23:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:23:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:23:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:23:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142144283mirror.1smithi11519.1.1-330-gc2ddf972falseOK42184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOK44314434mirror.0smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":17892352.0,"bytes_per_snapshot":536748032.0,"last_snapshot_bytes":536770560,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622582,"remote_snapshot_timestamp":1726622582,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:23:11cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:23:11' 2024-09-18T01:23:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:23:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:23:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:23:11' 2024-09-18T01:23:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:23:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:23:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:23:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:23:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:23:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap6 2024-09-18T01:23:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap6 2024-09-18T01:23:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap6 2024-09-18T01:23:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:23:33.808 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.32863526263473175) > weight (0.25) 2024-09-18T01:23:33.808 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-18T01:23:33.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-18T01:23:33.809 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-18T01:23:33.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:23:33.809+0000 7f9bfbb37640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 38344) UID: 1000 2024-09-18T01:23:40.809 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-18T01:23:40.810 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:23:40.810 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-18T01:23:40.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-18T01:23:40.810 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-18T01:23:40.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-18T01:23:40.814 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-18T01:23:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:23:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:23:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:23:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:23:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:23:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:23:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:23:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:23:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:23:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:23:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:23:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:23:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:23:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:23:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:23:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:23:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:23:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:23:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:23:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:23:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:23:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:23:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:23:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:23:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:23:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:23:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:23:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:23:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:23:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:23:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:23:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:23:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:23:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:23:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:23:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:23:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:23:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:23:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:23:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:23:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:23:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:23:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:23:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:23:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:23:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:23:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:23:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:23:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:23:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:23:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:23:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:23:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:23:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:23:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:23:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:23:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 20 2024-09-18T01:23:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:23:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:23:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:23:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:23:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:23:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=20 2024-09-18T01:23:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:23:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:23:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:23:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:23:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:23:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:23:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:23:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:23:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 20 = '' 2024-09-18T01:23:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:23:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:23:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:23:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:23:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:23:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:23:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:23:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:23:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:23:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 20 = '' 2024-09-18T01:23:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:23:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:23:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:23:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:23:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:23:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:23:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:23:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:23:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:23:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 20 = '' 2024-09-18T01:23:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:23:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:23:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:23:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:23:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:23:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=20 2024-09-18T01:23:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 20 = 20 2024-09-18T01:23:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-18T01:23:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap6 2024-09-18T01:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:23:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-18T01:23:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:23:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 12 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:22:12 2024 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:23:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:23:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:23:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:23:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:23:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:23:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap5-clean ']' 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap5-clean 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:24:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:24:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:24:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap6 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap6 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:24:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:24:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:24:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap6-clean 2024-09-18T01:24:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-18T01:24:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap6-clean 2024-09-18T01:24:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:24:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:24:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:24:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:24:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=12 2024-09-18T01:24:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:24:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:24:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 29991936 bytes 2024-09-18T01:24:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 52857856 bytes 2024-09-18T01:24:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 91526144 bytes 2024-09-18T01:24:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 109714432 bytes 2024-09-18T01:24:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 188680192 bytes 2024-09-18T01:24:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 194580480 bytes 2024-09-18T01:24:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 241933312 bytes 2024-09-18T01:24:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 342107136 bytes 2024-09-18T01:24:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 368854016 bytes 2024-09-18T01:24:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 417312768 bytes 2024-09-18T01:24:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 481158144 bytes 2024-09-18T01:24:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 528059392 bytes 2024-09-18T01:24:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 573950976 bytes 2024-09-18T01:24:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 622742528 bytes 2024-09-18T01:24:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 662608896 bytes 2024-09-18T01:24:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 680226816 bytes 2024-09-18T01:24:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 680993792 bytes 2024-09-18T01:24:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 724038656 bytes 2024-09-18T01:24:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 734338048 bytes 2024-09-18T01:24:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 760762368 bytes 2024-09-18T01:24:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 787037184 bytes 2024-09-18T01:24:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 812028928 bytes 2024-09-18T01:24:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 883361792 bytes 2024-09-18T01:24:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 943918080 bytes 2024-09-18T01:24:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 972426240 bytes 2024-09-18T01:24:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1007906816 bytes 2024-09-18T01:24:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1036524544 bytes 2024-09-18T01:24:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1093782528 bytes 2024-09-18T01:24:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1155637248 bytes 2024-09-18T01:24:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1243333632 bytes 2024-09-18T01:24:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1278565376 bytes 2024-09-18T01:24:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1321733120 bytes 2024-09-18T01:24:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1361491968 bytes 2024-09-18T01:24:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1407139840 bytes 2024-09-18T01:24:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1497091072 bytes 2024-09-18T01:24:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1522009088 bytes 2024-09-18T01:24:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1595139072 bytes 2024-09-18T01:24:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1655047168 bytes 2024-09-18T01:24:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1677510656 bytes 2024-09-18T01:24:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1741463552 bytes 2024-09-18T01:24:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1784373248 bytes 2024-09-18T01:24:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1806007296 bytes 2024-09-18T01:24:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1863432192 bytes 2024-09-18T01:24:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1918045184 bytes 2024-09-18T01:24:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1943145472 bytes 2024-09-18T01:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1954916352 bytes 2024-09-18T01:24:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2040568832 bytes 2024-09-18T01:24:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2054176768 bytes 2024-09-18T01:24:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2119439360 bytes 2024-09-18T01:24:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2155399168 bytes 2024-09-18T01:24:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2225488896 bytes 2024-09-18T01:24:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2260464640 bytes 2024-09-18T01:24:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2312984576 bytes 2024-09-18T01:24:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2313609216 bytes 2024-09-18T01:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2345789440 bytes 2024-09-18T01:24:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2373233664 bytes 2024-09-18T01:24:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2373825536 bytes 2024-09-18T01:24:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2407961600 bytes 2024-09-18T01:24:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2468566016 bytes 2024-09-18T01:24:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2504417280 bytes 2024-09-18T01:24:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2519843840 bytes 2024-09-18T01:24:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2539378688 bytes 2024-09-18T01:24:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2576436224 bytes 2024-09-18T01:24:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2595144704 bytes 2024-09-18T01:24:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2612864000 bytes 2024-09-18T01:24:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2638600192 bytes 2024-09-18T01:24:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2698441728 bytes 2024-09-18T01:24:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2709953536 bytes 2024-09-18T01:24:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2774334464 bytes 2024-09-18T01:24:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2808616960 bytes 2024-09-18T01:24:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2875557888 bytes 2024-09-18T01:24:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2922660864 bytes 2024-09-18T01:24:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2940108800 bytes 2024-09-18T01:24:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3011546112 bytes 2024-09-18T01:24:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3042709504 bytes 2024-09-18T01:24:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3075365888 bytes 2024-09-18T01:24:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3108812800 bytes 2024-09-18T01:24:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3113129984 bytes 2024-09-18T01:24:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3132086272 bytes 2024-09-18T01:24:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3203912704 bytes 2024-09-18T01:24:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3255729152 bytes 2024-09-18T01:24:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3266023424 bytes 2024-09-18T01:24:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3319564288 bytes 2024-09-18T01:24:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3377566720 bytes 2024-09-18T01:24:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3403426816 bytes 2024-09-18T01:24:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3410895872 bytes 2024-09-18T01:24:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3427458048 bytes 2024-09-18T01:24:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3453690880 bytes 2024-09-18T01:24:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3472730112 bytes 2024-09-18T01:24:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3490974720 bytes 2024-09-18T01:24:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3527563264 bytes 2024-09-18T01:24:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3537448960 bytes 2024-09-18T01:24:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 3586601984 bytes 2024-09-18T01:24:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 3608477696 bytes 2024-09-18T01:24:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 3647525888 bytes 2024-09-18T01:24:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 3719841792 bytes 2024-09-18T01:24:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 3788721152 bytes 2024-09-18T01:24:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 3795679232 bytes 2024-09-18T01:24:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 3814272000 bytes 2024-09-18T01:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 3852069888 bytes 2024-09-18T01:24:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 3871392768 bytes 2024-09-18T01:24:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 3920381952 bytes 2024-09-18T01:24:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 3946398720 bytes 2024-09-18T01:24:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 3998085120 bytes 2024-09-18T01:24:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4085176320 bytes 2024-09-18T01:24:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 4091101184 bytes 2024-09-18T01:24:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 4137821184 bytes 2024-09-18T01:24:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 4176428032 bytes 2024-09-18T01:24:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 4210114560 bytes 2024-09-18T01:24:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11000 IOs, 4243112960 bytes 2024-09-18T01:24:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11100 IOs, 4259614720 bytes 2024-09-18T01:24:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11200 IOs, 4318323712 bytes 2024-09-18T01:24:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11300 IOs, 4325707776 bytes 2024-09-18T01:24:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11400 IOs, 4381407232 bytes 2024-09-18T01:24:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11500 IOs, 4395069440 bytes 2024-09-18T01:24:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11600 IOs, 4440847360 bytes 2024-09-18T01:24:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11700 IOs, 4482804736 bytes 2024-09-18T01:24:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11800 IOs, 4526124032 bytes 2024-09-18T01:24:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:24:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:24:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:24:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:24:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:24:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:24:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:24:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:24:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:24:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:24:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:24:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":17892864.0,"bytes_per_snapshot":536778240.0,"last_snapshot_bytes":536785920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622634,"remote_snapshot_timestamp":1726622634,"replay_state":"idle"} 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:24:11 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:24:11' 2024-09-18T01:24:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:24:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:24:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:24:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":17892864.0,"bytes_per_snapshot":536778240.0,"last_snapshot_bytes":536785920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622634,"remote_snapshot_timestamp":1726622634,"replay_state":"idle"} 2024-09-18T01:24:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:24:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:24:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:24:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:24:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:24:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:24:11' 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:24:11' 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:24:11' 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:24:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142144283mirror.1smithi11519.1.1-330-gc2ddf972falseOK42184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOK44314434mirror.0smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":17892864.0,"bytes_per_snapshot":536778240.0,"last_snapshot_bytes":536785920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622634,"remote_snapshot_timestamp":1726622634,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:24:11cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:24:11' 2024-09-18T01:24:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:24:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:24:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:24:11' 2024-09-18T01:24:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:24:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:24:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:24:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:24:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:24:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap7 2024-09-18T01:24:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap7 2024-09-18T01:24:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap7 2024-09-18T01:24:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:24:23.698 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3337098635619782) > weight (0.25) 2024-09-18T01:24:23.699 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-18T01:24:23.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-18T01:24:23.699 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-18T01:24:23.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:24:23.699+0000 7fd7677f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 36732) UID: 1000 2024-09-18T01:24:25.700 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-18T01:24:25.700 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:24:25.701 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-18T01:24:25.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-18T01:24:25.701 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-18T01:24:25.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-18T01:24:25.704 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-18T01:24:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:24:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:24:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:24:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:24:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:24:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:24:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:24:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:24:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:24:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:24:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:24:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:24:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:24:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:24:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:24:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:24:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:24:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:24:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:24:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:24:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:24:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:24:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:24:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:24:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:24:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:24:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:24:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:24:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:24:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:24:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:24:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:24:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:24:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:24:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:24:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:24:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:24:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:24:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:24:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:24:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:24:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:24:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:24:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:24:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:24:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:24:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:24:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:24:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:24:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:24:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:24:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:24:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:24:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:24:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:24:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:24:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:24:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:24:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:24:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:24:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:24:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:24:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:24:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:24:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:24:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:24:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:24:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:24:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:24:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:24:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:24:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:24:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:24:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:24:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:24:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:24:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:24:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:24:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:24:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:24:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:24:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:24:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 23 2024-09-18T01:24:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:24:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:24:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:24:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:24:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:24:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=23 2024-09-18T01:24:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:24:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:24:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:24:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:24:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:24:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:24:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:24:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:24:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:24:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:24:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:24:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 23 = '' 2024-09-18T01:24:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:24:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:24:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:24:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:24:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:24:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:24:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:24:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:24:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:24:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:24:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 23 = '' 2024-09-18T01:24:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:24:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:24:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:24:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:24:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:24:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:24:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:24:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:24:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:24:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:24:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 23 = '' 2024-09-18T01:24:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:24:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:24:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:24:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:24:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:24:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:24:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=23 2024-09-18T01:24:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 23 = 23 2024-09-18T01:24:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-18T01:24:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:24:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap7 2024-09-18T01:24:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:24:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:24:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:24:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-18T01:24:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:24:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:24:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:24:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 14 2024-09-18T01:24:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:24:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:24:00 2024 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:24:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:24:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap6-clean ']' 2024-09-18T01:24:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-18T01:24:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap6-clean 2024-09-18T01:24:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:24:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:24:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:24:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:24:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:24:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap7 2024-09-18T01:24:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap7 2024-09-18T01:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:24:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:24:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:24:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:24:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:24:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap7-clean 2024-09-18T01:24:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-18T01:24:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap7-clean 2024-09-18T01:24:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:24:58.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:24:58.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:24:58.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:24:58.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:24:58.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:24:58.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:24:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:24:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:24:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:24:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 48653312 bytes 2024-09-18T01:24:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 82386944 bytes 2024-09-18T01:24:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 166149120 bytes 2024-09-18T01:24:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 191709184 bytes 2024-09-18T01:24:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 208331776 bytes 2024-09-18T01:24:58.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.836922491176417) > weight (0.25) 2024-09-18T01:24:58.815 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-18T01:24:58.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-18T01:24:58.815 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-18T01:24:58.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:24:58.816+0000 7f88aafa9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41028) UID: 1000 2024-09-18T01:24:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 218513408 bytes 2024-09-18T01:24:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 320027648 bytes 2024-09-18T01:24:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 429542400 bytes 2024-09-18T01:24:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 464360448 bytes 2024-09-18T01:24:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 488003584 bytes 2024-09-18T01:24:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 551828480 bytes 2024-09-18T01:24:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 583787520 bytes 2024-09-18T01:24:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 605941760 bytes 2024-09-18T01:24:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 632456192 bytes 2024-09-18T01:24:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 662161408 bytes 2024-09-18T01:25:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 695279616 bytes 2024-09-18T01:25:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 755765248 bytes 2024-09-18T01:25:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 798464000 bytes 2024-09-18T01:25:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 817507328 bytes 2024-09-18T01:25:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 890337280 bytes 2024-09-18T01:25:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 988980224 bytes 2024-09-18T01:25:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 996399104 bytes 2024-09-18T01:25:00.816 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-18T01:25:00.816 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:25:00.817 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-18T01:25:00.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-18T01:25:00.817 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-18T01:25:00.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-18T01:25:00.820 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-09-18T01:25:00.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1030497280 bytes 2024-09-18T01:25:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1059723264 bytes 2024-09-18T01:25:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1076026368 bytes 2024-09-18T01:25:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1138377728 bytes 2024-09-18T01:25:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1154194432 bytes 2024-09-18T01:25:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1154824192 bytes 2024-09-18T01:25:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1204094976 bytes 2024-09-18T01:25:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1228895232 bytes 2024-09-18T01:25:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1305534464 bytes 2024-09-18T01:25:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1352083456 bytes 2024-09-18T01:25:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1405089792 bytes 2024-09-18T01:25:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1495055360 bytes 2024-09-18T01:25:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1507994624 bytes 2024-09-18T01:25:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1553312768 bytes 2024-09-18T01:25:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1587341312 bytes 2024-09-18T01:25:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1617345536 bytes 2024-09-18T01:25:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1673575424 bytes 2024-09-18T01:25:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1722081280 bytes 2024-09-18T01:25:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1727834112 bytes 2024-09-18T01:25:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1737703424 bytes 2024-09-18T01:25:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1756876800 bytes 2024-09-18T01:25:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1894769664 bytes 2024-09-18T01:25:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1895409664 bytes 2024-09-18T01:25:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1934599168 bytes 2024-09-18T01:25:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1958411264 bytes 2024-09-18T01:25:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2007377920 bytes 2024-09-18T01:25:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2059808768 bytes 2024-09-18T01:25:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2090826752 bytes 2024-09-18T01:25:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2136249344 bytes 2024-09-18T01:25:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2143363072 bytes 2024-09-18T01:25:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:25:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:25:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:25:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:25:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:25:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:25:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:25:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536778240.0,"last_snapshot_bytes":536785920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622634,"remote_snapshot_timestamp":1726622634,"replay_state":"idle"} 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:24:41 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:24:41' 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:25:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:25:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:25:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536778240.0,"last_snapshot_bytes":536785920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622634,"remote_snapshot_timestamp":1726622634,"replay_state":"idle"} 2024-09-18T01:25:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:25:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:25:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:25:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:25:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:25:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:25:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:25:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:24:41' 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:24:41' 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:24:41' 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:25:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:25:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOK44314434mirror.0smithi11519.1.1-330-gc2ddf972falseOK45064509mirror.1smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536778240.0,"last_snapshot_bytes":536785920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622634,"remote_snapshot_timestamp":1726622634,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:24:41cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:24:41' 2024-09-18T01:25:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:25:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:25:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:24:41' 2024-09-18T01:25:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:25:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:25:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:25:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:25:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:25:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap8 2024-09-18T01:25:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap8 2024-09-18T01:25:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:25:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap8 2024-09-18T01:25:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:25:33.705 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5418667150871054) > weight (0.25) 2024-09-18T01:25:33.705 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.34368087907016986) > weight (0.25) 2024-09-18T01:25:33.706 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9243331515787502) > weight (0.25) 2024-09-18T01:25:33.706 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.41325050801197016) > weight (0.25) 2024-09-18T01:25:33.706 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:25:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:25:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:25:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:25:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:25:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:25:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:25:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:25:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:25:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:25:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:25:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:25:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:25:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:25:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:25:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:25:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:25:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:25:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:25:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:25:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:25:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:25:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:25:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:25:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:25:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:25:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:25:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:25:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:25:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:25:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:25:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:25:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:25:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:25:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:25:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:25:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:25:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:25:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:25:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:25:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:25:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:25:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:25:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:25:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:25:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:25:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:25:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:25:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:25:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:25:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:25:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:25:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:25:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:25:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:25:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:25:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:25:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:25:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:25:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:25:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:25:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 26 2024-09-18T01:25:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:25:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:25:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:25:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:25:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:25:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:25:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=26 2024-09-18T01:25:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:25:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:25:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:25:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:25:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:25:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:25:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:25:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:25:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:25:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:25:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:25:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 26 = '' 2024-09-18T01:25:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:25:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:25:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:25:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:25:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:25:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:25:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:25:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:25:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:25:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:25:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 26 = '' 2024-09-18T01:25:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:25:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:25:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:25:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:25:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:25:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:25:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:25:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:25:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:25:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:25:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 26 = '' 2024-09-18T01:25:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:25:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:25:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:25:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:25:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:25:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=26 2024-09-18T01:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 26 = 26 2024-09-18T01:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-18T01:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap8 2024-09-18T01:25:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:25:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:25:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:25:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 16 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:24:00 2024 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:25:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap7-clean ']' 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap7-clean 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:25:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:25:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:25:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap8 2024-09-18T01:25:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap8 2024-09-18T01:25:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:25:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:25:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:25:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:25:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:25:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:25:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap8-clean 2024-09-18T01:25:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-18T01:25:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:25:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap8-clean 2024-09-18T01:25:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:25:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:25:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:25:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:25:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:25:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:25:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:25:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:25:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 36301824 bytes 2024-09-18T01:25:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 88632320 bytes 2024-09-18T01:25:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 125395968 bytes 2024-09-18T01:25:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 228349952 bytes 2024-09-18T01:25:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 228960256 bytes 2024-09-18T01:25:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 254157824 bytes 2024-09-18T01:25:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 269518848 bytes 2024-09-18T01:25:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 339856384 bytes 2024-09-18T01:25:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 444157952 bytes 2024-09-18T01:25:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 478621696 bytes 2024-09-18T01:25:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 513748992 bytes 2024-09-18T01:25:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 514357248 bytes 2024-09-18T01:25:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 541471744 bytes 2024-09-18T01:25:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 581391360 bytes 2024-09-18T01:25:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 635958272 bytes 2024-09-18T01:25:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 664788992 bytes 2024-09-18T01:25:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 689516544 bytes 2024-09-18T01:25:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 727894016 bytes 2024-09-18T01:25:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 749673472 bytes 2024-09-18T01:25:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 794818560 bytes 2024-09-18T01:25:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 855768064 bytes 2024-09-18T01:25:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 880979968 bytes 2024-09-18T01:25:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 892856320 bytes 2024-09-18T01:25:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 929881088 bytes 2024-09-18T01:25:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 993245184 bytes 2024-09-18T01:25:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 993851392 bytes 2024-09-18T01:25:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1053925376 bytes 2024-09-18T01:25:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1061225472 bytes 2024-09-18T01:25:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1083611136 bytes 2024-09-18T01:25:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1103617024 bytes 2024-09-18T01:25:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1189259264 bytes 2024-09-18T01:25:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1225221120 bytes 2024-09-18T01:25:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1238425600 bytes 2024-09-18T01:25:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1269388288 bytes 2024-09-18T01:25:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1294298112 bytes 2024-09-18T01:25:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1323697152 bytes 2024-09-18T01:25:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1324306432 bytes 2024-09-18T01:25:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1401928704 bytes 2024-09-18T01:25:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1430622208 bytes 2024-09-18T01:25:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1479288832 bytes 2024-09-18T01:25:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1479902208 bytes 2024-09-18T01:25:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1506359296 bytes 2024-09-18T01:25:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1512780800 bytes 2024-09-18T01:25:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1581175808 bytes 2024-09-18T01:25:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1628913664 bytes 2024-09-18T01:25:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1689970688 bytes 2024-09-18T01:25:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1744595968 bytes 2024-09-18T01:25:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1768401920 bytes 2024-09-18T01:25:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1820281856 bytes 2024-09-18T01:25:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1847530496 bytes 2024-09-18T01:25:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1897947136 bytes 2024-09-18T01:25:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1940129792 bytes 2024-09-18T01:25:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 1974064128 bytes 2024-09-18T01:25:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:25:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:25:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:25:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:25:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:25:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:25:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:25:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:25:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:25:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:25:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":6571076.27,"bytes_per_snapshot":531694592.0,"last_snapshot_bytes":526642176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622737,"remote_snapshot_timestamp":1726622737,"replay_state":"idle"} 2024-09-18T01:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:25:41 2024-09-18T01:25:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:25:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:25:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:25:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:25:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:25:41' 2024-09-18T01:25:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:25:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:25:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:25:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":6571076.27,"bytes_per_snapshot":531694592.0,"last_snapshot_bytes":526642176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622737,"remote_snapshot_timestamp":1726622737,"replay_state":"idle"} 2024-09-18T01:25:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:25:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:25:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:25:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:25:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:25:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:25:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:25:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:25:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:25:41' 2024-09-18T01:25:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:25:41' 2024-09-18T01:25:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:25:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:25:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:25:41' 2024-09-18T01:25:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:25:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:25:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:25:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOK44314434mirror.0smithi11519.1.1-330-gc2ddf972falseOK45064509mirror.1smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":6571076.27,"bytes_per_snapshot":531694592.0,"last_snapshot_bytes":526642176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622737,"remote_snapshot_timestamp":1726622737,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:25:41cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:25:41' 2024-09-18T01:25:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:25:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:25:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:25:41' 2024-09-18T01:25:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:25:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:25:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:25:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:25:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:25:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:25:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap9 2024-09-18T01:25:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap9 2024-09-18T01:25:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:25:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:25:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:25:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap9 2024-09-18T01:25:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:26:13.820 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-18T01:26:13.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-18T01:26:13.822 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-18T01:26:13.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:26:13.822+0000 7f1606cd2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36829) UID: 1000 2024-09-18T01:26:18.822 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-18T01:26:18.822 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:26:18.822 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-18T01:26:18.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-18T01:26:18.823 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-18T01:26:18.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-18T01:26:18.826 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-09-18T01:26:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:26:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:26:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:26:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:26:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:26:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:26:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:26:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:26:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:26:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:26:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:26:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:26:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:26:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:26:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:26:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:26:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:26:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:26:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:26:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:26:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:26:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:26:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:26:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:26:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:26:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:26:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:26:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:26:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:26:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:26:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:26:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:26:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:26:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:26:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:26:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:26:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:26:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:26:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:26:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:26:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:26:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:26:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:26:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:26:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:26:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:26:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:26:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:26:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:26:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:26:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:26:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:26:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:26:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:26:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:26:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:26:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:26:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:26:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:26:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:26:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:26:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:26:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:26:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:26:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:26:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:26:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:26:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:26:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 29 2024-09-18T01:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:26:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:26:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=29 2024-09-18T01:26:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:26:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:26:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:26:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:26:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:26:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:26:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:26:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:26:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:26:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:26:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:26:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 29 = '' 2024-09-18T01:26:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:26:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:26:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:26:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:26:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:26:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:26:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 29 = '' 2024-09-18T01:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:26:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:26:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:26:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:26:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:26:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:26:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:26:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:26:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:26:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 29 = '' 2024-09-18T01:26:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:26:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:26:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:26:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:26:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:26:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=29 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 29 = 29 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap9 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:26:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:26:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-18T01:26:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:26:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:26:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 18 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:25:43 2024 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:26:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:26:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:26:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:26:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap8-clean ']' 2024-09-18T01:26:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-18T01:26:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap8-clean 2024-09-18T01:26:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:26:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:26:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:26:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:26:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:26:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap9 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap9 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:26:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:26:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:26:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap9-clean 2024-09-18T01:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-18T01:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap9-clean 2024-09-18T01:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:26:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:26:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror test 2024-09-18T01:26:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:26:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:26:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=14 2024-09-18T01:26:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:26:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-18T01:26:35.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 29104128 bytes 2024-09-18T01:26:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 67442688 bytes 2024-09-18T01:26:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 97589248 bytes 2024-09-18T01:26:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 114833408 bytes 2024-09-18T01:26:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 179581952 bytes 2024-09-18T01:26:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 236324864 bytes 2024-09-18T01:26:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 311866368 bytes 2024-09-18T01:26:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 370596864 bytes 2024-09-18T01:26:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 384214016 bytes 2024-09-18T01:26:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 445340672 bytes 2024-09-18T01:26:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 479881216 bytes 2024-09-18T01:26:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 515111936 bytes 2024-09-18T01:26:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 561937408 bytes 2024-09-18T01:26:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 650006528 bytes 2024-09-18T01:26:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 683641856 bytes 2024-09-18T01:26:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 803519488 bytes 2024-09-18T01:26:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 894544896 bytes 2024-09-18T01:26:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 961861632 bytes 2024-09-18T01:26:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 1014298624 bytes 2024-09-18T01:26:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1058038784 bytes 2024-09-18T01:26:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1074290688 bytes 2024-09-18T01:26:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1122350080 bytes 2024-09-18T01:26:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1140446208 bytes 2024-09-18T01:26:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1207389184 bytes 2024-09-18T01:26:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1243234304 bytes 2024-09-18T01:26:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1280825344 bytes 2024-09-18T01:26:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1281456128 bytes 2024-09-18T01:26:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1334927360 bytes 2024-09-18T01:26:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1370127360 bytes 2024-09-18T01:26:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1370732544 bytes 2024-09-18T01:26:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1483246592 bytes 2024-09-18T01:26:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1567892480 bytes 2024-09-18T01:26:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1646117888 bytes 2024-09-18T01:26:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1670944768 bytes 2024-09-18T01:26:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1708609536 bytes 2024-09-18T01:26:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1773603840 bytes 2024-09-18T01:26:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1840901120 bytes 2024-09-18T01:26:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1854186496 bytes 2024-09-18T01:26:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1901543424 bytes 2024-09-18T01:26:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1998552064 bytes 2024-09-18T01:26:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 2013143040 bytes 2024-09-18T01:26:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2080720896 bytes 2024-09-18T01:26:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2103626752 bytes 2024-09-18T01:26:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2168891392 bytes 2024-09-18T01:26:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2217520128 bytes 2024-09-18T01:26:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2257211392 bytes 2024-09-18T01:26:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2278904832 bytes 2024-09-18T01:26:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2363543552 bytes 2024-09-18T01:26:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2475168768 bytes 2024-09-18T01:26:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2550607872 bytes 2024-09-18T01:26:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2656224256 bytes 2024-09-18T01:26:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2771713024 bytes 2024-09-18T01:26:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2772316160 bytes 2024-09-18T01:26:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2790565888 bytes 2024-09-18T01:26:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2850534400 bytes 2024-09-18T01:26:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2917159936 bytes 2024-09-18T01:26:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2929768448 bytes 2024-09-18T01:26:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2950198272 bytes 2024-09-18T01:26:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2990657536 bytes 2024-09-18T01:26:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 3021305856 bytes 2024-09-18T01:26:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 3056172032 bytes 2024-09-18T01:26:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 3069791232 bytes 2024-09-18T01:26:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 3121337344 bytes 2024-09-18T01:26:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 3211231232 bytes 2024-09-18T01:26:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 3288826880 bytes 2024-09-18T01:26:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 3309092864 bytes 2024-09-18T01:26:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3363053568 bytes 2024-09-18T01:26:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3388298240 bytes 2024-09-18T01:26:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3389267968 bytes 2024-09-18T01:26:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3396813824 bytes 2024-09-18T01:26:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3521720320 bytes 2024-09-18T01:26:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3528544256 bytes 2024-09-18T01:26:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3565716480 bytes 2024-09-18T01:26:43.707 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.540740118129862) > weight (0.25) 2024-09-18T01:26:43.707 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6861996898876384) > weight (0.25) 2024-09-18T01:26:43.707 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4478675853080397) > weight (0.25) 2024-09-18T01:26:43.707 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3018128994844844) > weight (0.25) 2024-09-18T01:26:43.707 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-18T01:26:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3622804480 bytes 2024-09-18T01:26:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3699381248 bytes 2024-09-18T01:26:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3752540160 bytes 2024-09-18T01:26:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3773898752 bytes 2024-09-18T01:26:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3829258240 bytes 2024-09-18T01:26:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3847647232 bytes 2024-09-18T01:26:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3879947264 bytes 2024-09-18T01:26:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3932225536 bytes 2024-09-18T01:26:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3932827648 bytes 2024-09-18T01:26:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3975526400 bytes 2024-09-18T01:26:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3977089024 bytes 2024-09-18T01:26:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3982294016 bytes 2024-09-18T01:26:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3996906496 bytes 2024-09-18T01:26:45.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 4000749568 bytes 2024-09-18T01:26:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 4001345536 bytes 2024-09-18T01:26:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 4019050496 bytes 2024-09-18T01:26:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 4049128448 bytes 2024-09-18T01:26:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4117536768 bytes 2024-09-18T01:26:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4170620928 bytes 2024-09-18T01:26:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4221932544 bytes 2024-09-18T01:26:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4244055040 bytes 2024-09-18T01:26:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4304507904 bytes 2024-09-18T01:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4326395904 bytes 2024-09-18T01:26:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4339007488 bytes 2024-09-18T01:26:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4365306880 bytes 2024-09-18T01:26:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4391739392 bytes 2024-09-18T01:26:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4415506432 bytes 2024-09-18T01:26:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4428959744 bytes 2024-09-18T01:26:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4492453888 bytes 2024-09-18T01:26:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4531272704 bytes 2024-09-18T01:26:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 4588411904 bytes 2024-09-18T01:26:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4593146880 bytes 2024-09-18T01:26:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 4641562624 bytes 2024-09-18T01:26:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 4685594624 bytes 2024-09-18T01:26:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 4741415936 bytes 2024-09-18T01:26:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 4763960320 bytes 2024-09-18T01:26:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11000 IOs, 4785941504 bytes 2024-09-18T01:26:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11100 IOs, 4828591104 bytes 2024-09-18T01:26:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11200 IOs, 4881446912 bytes 2024-09-18T01:26:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11300 IOs, 4941700096 bytes 2024-09-18T01:26:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11400 IOs, 4989130752 bytes 2024-09-18T01:26:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11500 IOs, 5028043776 bytes 2024-09-18T01:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11600 IOs, 5067146240 bytes 2024-09-18T01:26:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11700 IOs, 5073595392 bytes 2024-09-18T01:26:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11800 IOs, 5128444928 bytes 2024-09-18T01:26:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11900 IOs, 5159378944 bytes 2024-09-18T01:26:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12000 IOs, 5203110912 bytes 2024-09-18T01:26:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12100 IOs, 5225426944 bytes 2024-09-18T01:26:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12200 IOs, 5265331200 bytes 2024-09-18T01:26:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:12300 IOs, 5301179392 bytes 2024-09-18T01:26:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:26:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local description_pattern= 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local service_pattern= 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='test: 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: global_id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+replaying 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: replaying, {"bytes_per_second":17353898.67,"bytes_per_snapshot":523629568.0,"last_snapshot_bytes":520616960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622786,"remote_snapshot_timestamp":1726622786,"replay_state":"idle"} 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: service: mirror.2 on smithi115 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:26:41 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: peer_sites: 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: name: cluster2 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: state: up+stopped 2024-09-18T01:26:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: description: local image is primary 2024-09-18T01:26:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: last_update: 2024-09-18 01:26:41' 2024-09-18T01:26:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T01:26:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: state: up+replaying 2024-09-18T01:26:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^ description: .*' 2024-09-18T01:26:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: description: replaying, {"bytes_per_second":17353898.67,"bytes_per_snapshot":523629568.0,"last_snapshot_bytes":520616960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622786,"remote_snapshot_timestamp":1726622786,"replay_state":"idle"} 2024-09-18T01:26:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:26:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:26:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep '^up+' 2024-09-18T01:26:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:26:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T01:26:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: service: mirror.2 on smithi115 2024-09-18T01:26:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update 2024-09-18T01:26:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T01:26:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:26:41' 2024-09-18T01:26:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-18 01:26:41' 2024-09-18T01:26:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:26:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state_pattern=up+replaying 2024-09-18T01:26:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local 'prev_last_update=2024-09-18 01:26:41' 2024-09-18T01:26:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status 2024-09-18T01:26:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ CEPH_ARGS= 2024-09-18T01:26:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T01:26:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status='OKOKOK142184277mirror.2smithi11519.1.1-330-gc2ddf972trueOK42224278mirror.3smithi11519.1.1-330-gc2ddf972falseOK44314434mirror.0smithi11519.1.1-330-gc2ddf972falseOK45064509mirror.1smithi11519.1.1-330-gc2ddf972falseOKtest3aa74b3d-0c1f-407a-a9b4-3a502f7f385aup+replayingreplaying, {"bytes_per_second":17353898.67,"bytes_per_snapshot":523629568.0,"last_snapshot_bytes":520616960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726622786,"remote_snapshot_timestamp":1726622786,"replay_state":"idle"}42184277mirror.2smithi1152024-09-18 01:26:41cluster2909fcd80-1380-4b17-b581-43d9e4cb4ecaup+stoppedlocal image is primary2024-09-18 01:26:41' 2024-09-18T01:26:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_update state 2024-09-18T01:26:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-18T01:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ last_update='2024-09-18 01:26:41' 2024-09-18T01:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-18T01:26:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=up+replaying 2024-09-18T01:26:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo up+replaying 2024-09-18T01:26:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep up+replaying 2024-09-18T01:26:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:up+replaying 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap10 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap10 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap10 2024-09-18T01:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:27:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:27:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:27:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:27:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:27:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:27:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:27:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:27:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:27:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:27:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:27:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:27:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:27:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:27:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:27:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:27:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:27:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:27:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:27:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:27:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:27:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:27:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:27:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:27:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:27:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:27:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:27:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:27:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:27:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:27:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:27:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:27:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:27:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:27:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:27:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:27:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:27:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:27:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:27:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:27:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:27:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:27:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:27:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:27:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:27:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:27:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:27:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:27:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:27:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:27:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:27:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:27:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:27:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:27:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:27:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:27:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:27:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:27:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:27:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:27:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:27:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:27:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:27:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 32 2024-09-18T01:27:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:27:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:27:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:27:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:27:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:27:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=32 2024-09-18T01:27:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:27:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:27:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:27:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:27:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:27:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:27:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:27:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:27:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:27:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:27:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:27:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 32 = '' 2024-09-18T01:27:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:27:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:27:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:27:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:27:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:27:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:27:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:27:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:27:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:27:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:27:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 32 = '' 2024-09-18T01:27:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:27:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:27:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:27:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:27:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:27:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:27:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:27:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:27:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:27:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:27:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 32 = '' 2024-09-18T01:27:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:27:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:27:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:27:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:27:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:27:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=32 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 32 = 32 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap10 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:27:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:27:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 20 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:27:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:25:43 2024 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:27:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snap9-clean ']' 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap9-clean 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:27.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap10 2024-09-18T01:27:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap10 2024-09-18T01:27:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ clean_snap_name=snap10-clean 2024-09-18T01:27:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-18T01:27:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap10-clean 2024-09-18T01:27:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-18T01:27:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-18T01:27:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-18T01:27:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:27:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:27:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:27:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:27:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-18T01:27:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:27:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:27:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:27:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:27:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:27:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:27:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:27:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-18T01:27:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:27:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:27:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:27:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:27:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:27:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:27:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:27:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:27:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:27:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:27:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:27:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-18T01:27:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:27:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:27:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:27:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:27:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-18T01:27:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:27:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:27:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:27:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:27:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:27:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:27:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:27:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:27:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:27:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:27:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:27:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-18T01:27:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-18T01:27:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:27:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:27:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:27:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:27:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:27:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:27:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:27:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-18T01:27:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:27:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:27:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:27:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:27:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-18T01:27:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:27:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-test.status 2024-09-18T01:27:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-test.status 2024-09-18T01:27:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:27:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:27:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-test.status 2024-09-18T01:27:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-test.status 2024-09-18T01:27:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:27:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-18T01:27:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-18T01:27:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 34 2024-09-18T01:27:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:27:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:27:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.status 2024-09-18T01:27:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-18T01:27:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:27:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:27:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=34 2024-09-18T01:27:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:27:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:27:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:27:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:27:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:27:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:27:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:27:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:27:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:27:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:27:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 34 = '' 2024-09-18T01:27:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:27:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:27:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.status 2024-09-18T01:27:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-18T01:27:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:27:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=34 2024-09-18T01:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 34 = 34 2024-09-18T01:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-18T01:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap10-clean 2024-09-18T01:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:27:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-18T01:27:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'test': 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 21 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5726f1b09 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5726f1b09 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:27:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:27:26 2024 2024-09-18T01:27:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:18:43 2024 2024-09-18T01:27:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:27:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:27:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:27:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3aa74b3d-0c1f-407a-a9b4-3a502f7f385a 2024-09-18T01:27:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:27:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 10 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap1 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap1 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap1 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap1-clean 2024-09-18T01:27:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-18T01:27:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap1-clean 2024-09-18T01:27:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap2 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap2 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap2 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap2-clean 2024-09-18T01:27:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-18T01:27:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap2-clean 2024-09-18T01:27:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap3 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap3 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap3 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap3-clean 2024-09-18T01:27:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-18T01:27:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap3-clean 2024-09-18T01:27:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap4 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap4 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap4 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:45.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap4-clean 2024-09-18T01:27:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-18T01:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap4-clean 2024-09-18T01:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap5 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap5 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap5 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap5-clean 2024-09-18T01:27:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-18T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap5-clean 2024-09-18T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap6 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap6 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap6 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap6-clean 2024-09-18T01:27:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-18T01:27:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap6-clean 2024-09-18T01:27:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap7 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap7 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap7 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap7-clean 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap7-clean 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap8 2024-09-18T01:27:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap8 2024-09-18T01:27:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap8 2024-09-18T01:27:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap8-clean 2024-09-18T01:27:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-18T01:27:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap8-clean 2024-09-18T01:27:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:27:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:27:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:27:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap9 2024-09-18T01:27:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap9 2024-09-18T01:27:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:27:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:27:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap9 2024-09-18T01:27:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:27:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:27:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:27:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:28:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:28:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:28:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap9-clean 2024-09-18T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-18T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap9-clean 2024-09-18T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:28:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:28:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:28:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap10 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap10 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap10 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:28:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:28:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:28:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap10-clean 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap10-clean 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:28:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export 2024-09-18T01:28:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:28:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:05.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3746882045767377) > weight (0.25) 2024-09-18T01:28:05.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.45473732758055385) > weight (0.25) 2024-09-18T01:28:05.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8925113616343352) > weight (0.25) 2024-09-18T01:28:05.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7060804051786105) > weight (0.25) 2024-09-18T01:28:05.827 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-09-18T01:28:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-18T01:28:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-test.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-test.export 2024-09-18T01:28:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 10 2024-09-18T01:28:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap1 2024-09-18T01:28:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-18T01:28:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap1 2024-09-18T01:28:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-18T01:28:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap1-clean 2024-09-18T01:28:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-18T01:28:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap1-clean 2024-09-18T01:28:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-18T01:28:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap2 2024-09-18T01:28:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-18T01:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap2 2024-09-18T01:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-18T01:28:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap2-clean 2024-09-18T01:28:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-18T01:28:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap2-clean 2024-09-18T01:28:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-18T01:28:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap3 2024-09-18T01:28:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-18T01:28:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap3 2024-09-18T01:28:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-18T01:28:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap3-clean 2024-09-18T01:28:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-18T01:28:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap3-clean 2024-09-18T01:28:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-18T01:28:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap4 2024-09-18T01:28:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-18T01:28:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap4 2024-09-18T01:28:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-18T01:28:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap4-clean 2024-09-18T01:28:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-18T01:28:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap4-clean 2024-09-18T01:28:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-18T01:28:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap5 2024-09-18T01:28:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-18T01:28:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap5 2024-09-18T01:28:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-18T01:28:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap5-clean 2024-09-18T01:28:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-18T01:28:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap5-clean 2024-09-18T01:28:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-18T01:28:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap6 2024-09-18T01:28:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-18T01:28:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap6 2024-09-18T01:28:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-18T01:28:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap6-clean 2024-09-18T01:28:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-18T01:28:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap6-clean 2024-09-18T01:28:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-18T01:28:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap7 2024-09-18T01:28:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-18T01:28:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap7 2024-09-18T01:28:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-18T01:28:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap7-clean 2024-09-18T01:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-18T01:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap7-clean 2024-09-18T01:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-18T01:28:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap8 2024-09-18T01:28:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-18T01:28:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap8 2024-09-18T01:28:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-18T01:28:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap8-clean 2024-09-18T01:28:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-18T01:28:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap8-clean 2024-09-18T01:28:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-18T01:28:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap9 2024-09-18T01:28:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-18T01:28:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap9 2024-09-18T01:28:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-18T01:28:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap9-clean 2024-09-18T01:28:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-18T01:28:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap9-clean 2024-09-18T01:28:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-18T01:28:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:28:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap10 2024-09-18T01:28:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-18T01:28:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap10 2024-09-18T01:28:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-18T01:28:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap10-clean 2024-09-18T01:28:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-18T01:28:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap10-clean 2024-09-18T01:28:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-18T01:28:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:28:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror test 2024-09-18T01:28:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:28:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:28:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror test 2024-09-18T01:28:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-18T01:28:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-18T01:28:26.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:28:26.340+0000 7f11b9a83640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5572f80b0000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:28:26.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:28:26.340+0000 7f11c0290640 -1 rbd::mirror::ImageReplayer: 0x5572f7e52280 [3/3aa74b3d-0c1f-407a-a9b4-3a502f7f385a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:28:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:28:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror test 2024-09-18T01:28:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:28:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:28:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=test 2024-09-18T01:28:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-18T01:28:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:28:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-18T01:28:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:28:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:28:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=test 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror test 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=test 2024-09-18T01:28:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-18T01:28:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:28:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-18T01:28:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:28:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:28:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:28:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ testlog 'TEST: create many images' 2024-09-18T01:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ tee -a /tmp/tmp.YxZgdY10x3/rbd-mirror.test.log 2024-09-18T01:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ date '+%F %T' 2024-09-18T01:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo 2024-09-18 01:28:28 TEST: create many images 2024-09-18T01:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:2024-09-18 01:28:28 TEST: create many images 2024-09-18T01:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ snap_name=snap 2024-09-18T01:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 50 2024-09-18T01:28:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:28:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_1 2024-09-18T01:28:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-18T01:28:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:28:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:28:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:28:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:28:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-18T01:28:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-18T01:28:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:28:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-18T01:28:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_1': 2024-09-18T01:28:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:28:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 1268fd613ab9 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.1268fd613ab9 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:28:28 2024 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:28:28 2024 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:28:28 2024 2024-09-18T01:28:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:28:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:28:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: e824ac62-3a6a-4a7a-9f9f-f4f31117d463 2024-09-18T01:28:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:28:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-18T01:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=10 2024-09-18T01:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:28:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 17840128 bytes 2024-09-18T01:28:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 95637504 bytes 2024-09-18T01:28:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 156782592 bytes 2024-09-18T01:28:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 234175488 bytes 2024-09-18T01:28:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 279775232 bytes 2024-09-18T01:28:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 280389632 bytes 2024-09-18T01:28:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 337841152 bytes 2024-09-18T01:28:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 445837312 bytes 2024-09-18T01:28:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 490558464 bytes 2024-09-18T01:28:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 588148736 bytes 2024-09-18T01:28:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 611416064 bytes 2024-09-18T01:28:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 694636544 bytes 2024-09-18T01:28:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 744105984 bytes 2024-09-18T01:28:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 802961408 bytes 2024-09-18T01:28:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 852677632 bytes 2024-09-18T01:28:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 899893248 bytes 2024-09-18T01:28:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 904327168 bytes 2024-09-18T01:28:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 910133248 bytes 2024-09-18T01:28:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 919823360 bytes 2024-09-18T01:28:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 927420416 bytes 2024-09-18T01:28:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 954725376 bytes 2024-09-18T01:28:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 993085440 bytes 2024-09-18T01:28:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1088108544 bytes 2024-09-18T01:28:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1151790080 bytes 2024-09-18T01:28:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1184461824 bytes 2024-09-18T01:28:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1227364352 bytes 2024-09-18T01:28:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1242580992 bytes 2024-09-18T01:28:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1312225280 bytes 2024-09-18T01:28:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1327336448 bytes 2024-09-18T01:28:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1400846336 bytes 2024-09-18T01:28:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1445053440 bytes 2024-09-18T01:28:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1485346816 bytes 2024-09-18T01:28:32.708 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-18T01:28:32.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-18T01:28:32.709 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-18T01:28:32.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:28:32.709+0000 7f7cbd432640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 40459) UID: 1000 2024-09-18T01:28:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1540319232 bytes 2024-09-18T01:28:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1572596736 bytes 2024-09-18T01:28:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1615147008 bytes 2024-09-18T01:28:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1643125760 bytes 2024-09-18T01:28:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1663127552 bytes 2024-09-18T01:28:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1695782912 bytes 2024-09-18T01:28:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1804376064 bytes 2024-09-18T01:28:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1871672320 bytes 2024-09-18T01:28:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1949068288 bytes 2024-09-18T01:28:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1993677824 bytes 2024-09-18T01:28:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1994284032 bytes 2024-09-18T01:28:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2069775360 bytes 2024-09-18T01:28:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2079934464 bytes 2024-09-18T01:28:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2104108032 bytes 2024-09-18T01:28:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2162601984 bytes 2024-09-18T01:28:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2192401408 bytes 2024-09-18T01:28:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2206292992 bytes 2024-09-18T01:28:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2231545856 bytes 2024-09-18T01:28:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2257540096 bytes 2024-09-18T01:28:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2260996096 bytes 2024-09-18T01:28:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2285682688 bytes 2024-09-18T01:28:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2297033728 bytes 2024-09-18T01:28:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2336903168 bytes 2024-09-18T01:28:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2398865408 bytes 2024-09-18T01:28:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2401740800 bytes 2024-09-18T01:28:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2439357440 bytes 2024-09-18T01:28:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2468751360 bytes 2024-09-18T01:28:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2530173952 bytes 2024-09-18T01:28:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2586733568 bytes 2024-09-18T01:28:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2646240256 bytes 2024-09-18T01:28:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2688787456 bytes 2024-09-18T01:28:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2703877120 bytes 2024-09-18T01:28:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2731122688 bytes 2024-09-18T01:28:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2774733824 bytes 2024-09-18T01:28:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2784326656 bytes 2024-09-18T01:28:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2950736896 bytes 2024-09-18T01:28:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3005802496 bytes 2024-09-18T01:28:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3068567552 bytes 2024-09-18T01:28:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3145610240 bytes 2024-09-18T01:28:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3181048832 bytes 2024-09-18T01:28:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3192954880 bytes 2024-09-18T01:28:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3262999552 bytes 2024-09-18T01:28:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3341851648 bytes 2024-09-18T01:28:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3362555904 bytes 2024-09-18T01:28:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3377423360 bytes 2024-09-18T01:28:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3419748352 bytes 2024-09-18T01:28:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3447390208 bytes 2024-09-18T01:28:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3487956992 bytes 2024-09-18T01:28:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3535404032 bytes 2024-09-18T01:28:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3560990720 bytes 2024-09-18T01:28:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3626450944 bytes 2024-09-18T01:28:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3669168128 bytes 2024-09-18T01:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3689407488 bytes 2024-09-18T01:28:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3729381376 bytes 2024-09-18T01:28:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3791549440 bytes 2024-09-18T01:28:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3815354368 bytes 2024-09-18T01:28:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3891574784 bytes 2024-09-18T01:28:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3948523520 bytes 2024-09-18T01:28:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3996171264 bytes 2024-09-18T01:28:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3999921152 bytes 2024-09-18T01:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4043059200 bytes 2024-09-18T01:28:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4081505280 bytes 2024-09-18T01:28:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4156269568 bytes 2024-09-18T01:28:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4179098624 bytes 2024-09-18T01:28:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:28:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:28:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:28:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:28:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_2 2024-09-18T01:28:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-18T01:28:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:28:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:28:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-18T01:28:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:28:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:28:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-18T01:28:39.710 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-18T01:28:39.710 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:28:39.711 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-18T01:28:39.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-18T01:28:39.711 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-18T01:28:39.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-18T01:28:39.714 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-18T01:28:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:28:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-18T01:28:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_2': 2024-09-18T01:28:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:28:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:28:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 127317955b4f 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.127317955b4f 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:28:39 2024 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:28:39 2024 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:28:39 2024 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:28:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 9dc75a63-a2d8-4ff1-bf1e-f7d6b7f28a2c 2024-09-18T01:28:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:28:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:28:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-18T01:28:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:28:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:28:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=10 2024-09-18T01:28:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:28:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:28:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 72958976 bytes 2024-09-18T01:28:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 176441344 bytes 2024-09-18T01:28:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 276613120 bytes 2024-09-18T01:28:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 311382016 bytes 2024-09-18T01:28:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 369673216 bytes 2024-09-18T01:28:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 427138048 bytes 2024-09-18T01:28:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 486876160 bytes 2024-09-18T01:28:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 540894208 bytes 2024-09-18T01:28:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 584112128 bytes 2024-09-18T01:28:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 618565632 bytes 2024-09-18T01:28:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 677014528 bytes 2024-09-18T01:28:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 713245696 bytes 2024-09-18T01:28:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 792946688 bytes 2024-09-18T01:28:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 854803456 bytes 2024-09-18T01:28:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 904639488 bytes 2024-09-18T01:28:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 914023424 bytes 2024-09-18T01:28:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 935293952 bytes 2024-09-18T01:28:42.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:28:42.117+0000 7ff677e02640 -1 rbd::mirror::ImageReplayer: 0x55572725a280 [3/9dc75a63-a2d8-4ff1-bf1e-f7d6b7f28a2c] start: already running 2024-09-18T01:28:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 1000404992 bytes 2024-09-18T01:28:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 1052253184 bytes 2024-09-18T01:28:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1128863744 bytes 2024-09-18T01:28:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1139637248 bytes 2024-09-18T01:28:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1192157184 bytes 2024-09-18T01:28:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1249878016 bytes 2024-09-18T01:28:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1269202944 bytes 2024-09-18T01:28:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1305408512 bytes 2024-09-18T01:28:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1332453376 bytes 2024-09-18T01:28:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1354840064 bytes 2024-09-18T01:28:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1361132544 bytes 2024-09-18T01:28:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1367899136 bytes 2024-09-18T01:28:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1372125184 bytes 2024-09-18T01:28:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1419016192 bytes 2024-09-18T01:28:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1472087040 bytes 2024-09-18T01:28:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1555508224 bytes 2024-09-18T01:28:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1563415552 bytes 2024-09-18T01:28:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1631534080 bytes 2024-09-18T01:28:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1691338752 bytes 2024-09-18T01:28:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1757372416 bytes 2024-09-18T01:28:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1810182144 bytes 2024-09-18T01:28:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1812666368 bytes 2024-09-18T01:28:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1834469376 bytes 2024-09-18T01:28:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1886649344 bytes 2024-09-18T01:28:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1904392192 bytes 2024-09-18T01:28:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1940397056 bytes 2024-09-18T01:28:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1971092480 bytes 2024-09-18T01:28:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2000420864 bytes 2024-09-18T01:28:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2062024704 bytes 2024-09-18T01:28:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2104748032 bytes 2024-09-18T01:28:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2190708736 bytes 2024-09-18T01:28:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2226333696 bytes 2024-09-18T01:28:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2226974720 bytes 2024-09-18T01:28:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2267683840 bytes 2024-09-18T01:28:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2340722688 bytes 2024-09-18T01:28:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2376365056 bytes 2024-09-18T01:28:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2394704896 bytes 2024-09-18T01:28:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2437124096 bytes 2024-09-18T01:28:45.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2449865728 bytes 2024-09-18T01:28:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2483652608 bytes 2024-09-18T01:28:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2555653120 bytes 2024-09-18T01:28:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2590169088 bytes 2024-09-18T01:28:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2619070464 bytes 2024-09-18T01:28:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2684774400 bytes 2024-09-18T01:28:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2717644800 bytes 2024-09-18T01:28:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2723305472 bytes 2024-09-18T01:28:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2745569280 bytes 2024-09-18T01:28:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2746188800 bytes 2024-09-18T01:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2816305152 bytes 2024-09-18T01:28:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2860609536 bytes 2024-09-18T01:28:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2935959552 bytes 2024-09-18T01:28:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2955736064 bytes 2024-09-18T01:28:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3010305024 bytes 2024-09-18T01:28:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3046113280 bytes 2024-09-18T01:28:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3123624960 bytes 2024-09-18T01:28:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3234521088 bytes 2024-09-18T01:28:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3282444288 bytes 2024-09-18T01:28:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3294801920 bytes 2024-09-18T01:28:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3335691264 bytes 2024-09-18T01:28:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3376215040 bytes 2024-09-18T01:28:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3395912704 bytes 2024-09-18T01:28:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3506333696 bytes 2024-09-18T01:28:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3524104192 bytes 2024-09-18T01:28:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3547419648 bytes 2024-09-18T01:28:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3565552640 bytes 2024-09-18T01:28:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3628349440 bytes 2024-09-18T01:28:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3733590016 bytes 2024-09-18T01:28:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3753379840 bytes 2024-09-18T01:28:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3795512320 bytes 2024-09-18T01:28:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3875376128 bytes 2024-09-18T01:28:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3887096832 bytes 2024-09-18T01:28:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3914083328 bytes 2024-09-18T01:28:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3941982208 bytes 2024-09-18T01:28:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3972975616 bytes 2024-09-18T01:28:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4016340992 bytes 2024-09-18T01:28:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4063273984 bytes 2024-09-18T01:28:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4156072960 bytes 2024-09-18T01:28:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4180696064 bytes 2024-09-18T01:28:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4204404736 bytes 2024-09-18T01:28:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:28:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:28:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:28:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:28:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_3 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:28:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:28:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:28:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:28:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-18T01:28:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:28:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:28:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-18T01:28:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:28:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:28:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-18T01:28:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:28:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-18T01:28:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_3': 2024-09-18T01:28:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:28:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 128532fd56ca 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.128532fd56ca 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:28:50 2024 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:28:50 2024 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:28:50 2024 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:28:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:28:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 4cad14e9-5f0f-48cb-acf6-96241971c0cd 2024-09-18T01:28:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-18T01:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:28:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:28:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:28:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:28:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 32681984 bytes 2024-09-18T01:28:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 64695296 bytes 2024-09-18T01:28:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 129038336 bytes 2024-09-18T01:28:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 158415872 bytes 2024-09-18T01:28:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 215118848 bytes 2024-09-18T01:28:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 281096192 bytes 2024-09-18T01:28:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 337720320 bytes 2024-09-18T01:28:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 404772864 bytes 2024-09-18T01:28:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 452916224 bytes 2024-09-18T01:28:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 457467904 bytes 2024-09-18T01:28:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 518131712 bytes 2024-09-18T01:28:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 531453952 bytes 2024-09-18T01:28:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 568506368 bytes 2024-09-18T01:28:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 583218176 bytes 2024-09-18T01:28:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 652262400 bytes 2024-09-18T01:28:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 711772160 bytes 2024-09-18T01:28:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 726257664 bytes 2024-09-18T01:28:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 807983104 bytes 2024-09-18T01:28:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 863427584 bytes 2024-09-18T01:28:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 893779968 bytes 2024-09-18T01:28:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 922009600 bytes 2024-09-18T01:28:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1009811456 bytes 2024-09-18T01:28:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1071232000 bytes 2024-09-18T01:28:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1081567232 bytes 2024-09-18T01:28:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1126711296 bytes 2024-09-18T01:28:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1217944576 bytes 2024-09-18T01:28:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1288386560 bytes 2024-09-18T01:28:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1352896512 bytes 2024-09-18T01:28:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1383435264 bytes 2024-09-18T01:28:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1433538560 bytes 2024-09-18T01:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1468506112 bytes 2024-09-18T01:28:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1529072640 bytes 2024-09-18T01:28:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1580508160 bytes 2024-09-18T01:28:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1603124224 bytes 2024-09-18T01:28:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1648240640 bytes 2024-09-18T01:28:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1734086656 bytes 2024-09-18T01:28:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1757076480 bytes 2024-09-18T01:28:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1803238400 bytes 2024-09-18T01:28:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1803821056 bytes 2024-09-18T01:28:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1816691712 bytes 2024-09-18T01:28:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1870442496 bytes 2024-09-18T01:28:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1923851264 bytes 2024-09-18T01:28:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1940119552 bytes 2024-09-18T01:28:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_4 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:28:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:28:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-18T01:28:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:28:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:28:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-18T01:28:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:28:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_4': 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 12911942cccb 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.12911942cccb 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:28:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:28:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:28:56 2024 2024-09-18T01:28:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:28:56 2024 2024-09-18T01:28:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:28:56 2024 2024-09-18T01:28:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:28:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:28:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 69e2ede3-6b24-471e-8e85-a7ce3370267a 2024-09-18T01:28:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:28:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:28:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-18T01:28:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:28:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:28:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:28:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:28:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=8 2024-09-18T01:28:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:28:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:28:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 53424128 bytes 2024-09-18T01:28:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 119136256 bytes 2024-09-18T01:28:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 145577984 bytes 2024-09-18T01:28:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 172660736 bytes 2024-09-18T01:28:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 217421824 bytes 2024-09-18T01:28:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 282660864 bytes 2024-09-18T01:28:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 301429760 bytes 2024-09-18T01:28:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 350991360 bytes 2024-09-18T01:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 366303232 bytes 2024-09-18T01:28:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 389713920 bytes 2024-09-18T01:28:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 408480768 bytes 2024-09-18T01:28:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 434768896 bytes 2024-09-18T01:28:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 456733696 bytes 2024-09-18T01:28:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 525434880 bytes 2024-09-18T01:28:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 549445632 bytes 2024-09-18T01:28:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 555351040 bytes 2024-09-18T01:28:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 580074496 bytes 2024-09-18T01:28:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 610785280 bytes 2024-09-18T01:28:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 612032512 bytes 2024-09-18T01:28:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 655722496 bytes 2024-09-18T01:28:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 685900800 bytes 2024-09-18T01:28:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 734248960 bytes 2024-09-18T01:28:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 753977344 bytes 2024-09-18T01:28:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 789073920 bytes 2024-09-18T01:28:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 809987072 bytes 2024-09-18T01:28:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 814970880 bytes 2024-09-18T01:28:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 839240704 bytes 2024-09-18T01:28:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 872631296 bytes 2024-09-18T01:29:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 930450432 bytes 2024-09-18T01:29:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 960308224 bytes 2024-09-18T01:29:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 997873664 bytes 2024-09-18T01:29:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1032514560 bytes 2024-09-18T01:29:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1041432576 bytes 2024-09-18T01:29:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1116910592 bytes 2024-09-18T01:29:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1156579328 bytes 2024-09-18T01:29:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1230760960 bytes 2024-09-18T01:29:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1267503104 bytes 2024-09-18T01:29:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1329410048 bytes 2024-09-18T01:29:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1392905216 bytes 2024-09-18T01:29:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1448524800 bytes 2024-09-18T01:29:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1517818880 bytes 2024-09-18T01:29:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1547157504 bytes 2024-09-18T01:29:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1624280064 bytes 2024-09-18T01:29:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1653615616 bytes 2024-09-18T01:29:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1659781120 bytes 2024-09-18T01:29:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1668623360 bytes 2024-09-18T01:29:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1750343680 bytes 2024-09-18T01:29:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1800028160 bytes 2024-09-18T01:29:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1824500736 bytes 2024-09-18T01:29:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1845630976 bytes 2024-09-18T01:29:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1899571200 bytes 2024-09-18T01:29:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1945943040 bytes 2024-09-18T01:29:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2008851456 bytes 2024-09-18T01:29:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2045993984 bytes 2024-09-18T01:29:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2097007616 bytes 2024-09-18T01:29:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2131154944 bytes 2024-09-18T01:29:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2153431040 bytes 2024-09-18T01:29:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2175939584 bytes 2024-09-18T01:29:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2225546240 bytes 2024-09-18T01:29:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2263915520 bytes 2024-09-18T01:29:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2267840512 bytes 2024-09-18T01:29:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2337603584 bytes 2024-09-18T01:29:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2408348672 bytes 2024-09-18T01:29:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2470400000 bytes 2024-09-18T01:29:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2504140800 bytes 2024-09-18T01:29:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2530173952 bytes 2024-09-18T01:29:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2584689664 bytes 2024-09-18T01:29:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2630876160 bytes 2024-09-18T01:29:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2723972096 bytes 2024-09-18T01:29:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2740678656 bytes 2024-09-18T01:29:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2839153664 bytes 2024-09-18T01:29:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2893154304 bytes 2024-09-18T01:29:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2960027648 bytes 2024-09-18T01:29:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 2985924608 bytes 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_5 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:29:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:29:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:29:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-18T01:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-18T01:29:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:29:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_5': 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 129d40de6940 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.129d40de6940 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:29:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:05 2024 2024-09-18T01:29:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:05 2024 2024-09-18T01:29:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:05 2024 2024-09-18T01:29:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:29:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:29:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: f65227d5-766c-440e-beb8-127249467502 2024-09-18T01:29:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:29:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:29:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-18T01:29:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:29:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:29:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=8 2024-09-18T01:29:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:29:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:29:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 34827264 bytes 2024-09-18T01:29:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 73667584 bytes 2024-09-18T01:29:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 117050368 bytes 2024-09-18T01:29:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 127695872 bytes 2024-09-18T01:29:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 158105600 bytes 2024-09-18T01:29:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 207773696 bytes 2024-09-18T01:29:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 309750784 bytes 2024-09-18T01:29:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 370365440 bytes 2024-09-18T01:29:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 412275712 bytes 2024-09-18T01:29:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 518916096 bytes 2024-09-18T01:29:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 562382848 bytes 2024-09-18T01:29:07.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 637786112 bytes 2024-09-18T01:29:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 683925504 bytes 2024-09-18T01:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 728457216 bytes 2024-09-18T01:29:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 796328960 bytes 2024-09-18T01:29:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 812143616 bytes 2024-09-18T01:29:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 845445120 bytes 2024-09-18T01:29:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 865843200 bytes 2024-09-18T01:29:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 886190080 bytes 2024-09-18T01:29:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 926649344 bytes 2024-09-18T01:29:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 994030592 bytes 2024-09-18T01:29:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1022123008 bytes 2024-09-18T01:29:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1112087552 bytes 2024-09-18T01:29:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1143717888 bytes 2024-09-18T01:29:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1189786624 bytes 2024-09-18T01:29:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1232833536 bytes 2024-09-18T01:29:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1279933440 bytes 2024-09-18T01:29:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1297047552 bytes 2024-09-18T01:29:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1360379904 bytes 2024-09-18T01:29:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1423185920 bytes 2024-09-18T01:29:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1487362048 bytes 2024-09-18T01:29:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1554519040 bytes 2024-09-18T01:29:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1624904704 bytes 2024-09-18T01:29:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1682604032 bytes 2024-09-18T01:29:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1708112896 bytes 2024-09-18T01:29:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1716879360 bytes 2024-09-18T01:29:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1766405120 bytes 2024-09-18T01:29:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1807839232 bytes 2024-09-18T01:29:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1826750464 bytes 2024-09-18T01:29:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1842850816 bytes 2024-09-18T01:29:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1933852672 bytes 2024-09-18T01:29:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2012912640 bytes 2024-09-18T01:29:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2038813696 bytes 2024-09-18T01:29:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2062373888 bytes 2024-09-18T01:29:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2132975616 bytes 2024-09-18T01:29:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2157361152 bytes 2024-09-18T01:29:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2249812992 bytes 2024-09-18T01:29:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2279117824 bytes 2024-09-18T01:29:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2331790336 bytes 2024-09-18T01:29:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2380257280 bytes 2024-09-18T01:29:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2459982848 bytes 2024-09-18T01:29:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2475875328 bytes 2024-09-18T01:29:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2501408768 bytes 2024-09-18T01:29:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2535650304 bytes 2024-09-18T01:29:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2543641600 bytes 2024-09-18T01:29:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2574256128 bytes 2024-09-18T01:29:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2619754496 bytes 2024-09-18T01:29:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2640658432 bytes 2024-09-18T01:29:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2676000768 bytes 2024-09-18T01:29:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2685409280 bytes 2024-09-18T01:29:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2728516608 bytes 2024-09-18T01:29:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2811577344 bytes 2024-09-18T01:29:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2899630080 bytes 2024-09-18T01:29:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2900245504 bytes 2024-09-18T01:29:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2989954048 bytes 2024-09-18T01:29:13.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 3035436032 bytes 2024-09-18T01:29:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3070249984 bytes 2024-09-18T01:29:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3089296384 bytes 2024-09-18T01:29:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3138246656 bytes 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_6 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:29:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:29:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:29:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:29:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-18T01:29:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-18T01:29:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:29:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-18T01:29:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_6': 2024-09-18T01:29:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:29:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:29:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:29:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 12a9ccbbf7ba 2024-09-18T01:29:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.12a9ccbbf7ba 2024-09-18T01:29:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:13 2024 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:13 2024 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:13 2024 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 72ab9ec5-f35c-428f-a831-18f79d59a14f 2024-09-18T01:29:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:29:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:29:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-18T01:29:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:29:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:29:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=12 2024-09-18T01:29:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:29:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:29:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 63002624 bytes 2024-09-18T01:29:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 159831040 bytes 2024-09-18T01:29:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 169591808 bytes 2024-09-18T01:29:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 203130880 bytes 2024-09-18T01:29:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 203745280 bytes 2024-09-18T01:29:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 216132608 bytes 2024-09-18T01:29:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 216753152 bytes 2024-09-18T01:29:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 271550464 bytes 2024-09-18T01:29:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 328553472 bytes 2024-09-18T01:29:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 357261312 bytes 2024-09-18T01:29:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 373572608 bytes 2024-09-18T01:29:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 412998656 bytes 2024-09-18T01:29:16.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 475680768 bytes 2024-09-18T01:29:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 493455360 bytes 2024-09-18T01:29:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 551385088 bytes 2024-09-18T01:29:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 604052480 bytes 2024-09-18T01:29:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 623938560 bytes 2024-09-18T01:29:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 638511104 bytes 2024-09-18T01:29:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 736375808 bytes 2024-09-18T01:29:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 770676736 bytes 2024-09-18T01:29:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 835903488 bytes 2024-09-18T01:29:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 884023296 bytes 2024-09-18T01:29:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 997378048 bytes 2024-09-18T01:29:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1073007616 bytes 2024-09-18T01:29:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1119516672 bytes 2024-09-18T01:29:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1120741376 bytes 2024-09-18T01:29:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1138544640 bytes 2024-09-18T01:29:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1178649600 bytes 2024-09-18T01:29:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1212168192 bytes 2024-09-18T01:29:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1259709440 bytes 2024-09-18T01:29:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1296014336 bytes 2024-09-18T01:29:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1387895808 bytes 2024-09-18T01:29:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1449686016 bytes 2024-09-18T01:29:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1491049472 bytes 2024-09-18T01:29:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1553181696 bytes 2024-09-18T01:29:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1627484160 bytes 2024-09-18T01:29:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1685542912 bytes 2024-09-18T01:29:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1755832320 bytes 2024-09-18T01:29:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1792462848 bytes 2024-09-18T01:29:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1863178240 bytes 2024-09-18T01:29:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1880266752 bytes 2024-09-18T01:29:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1964219392 bytes 2024-09-18T01:29:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2033833984 bytes 2024-09-18T01:29:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2072884224 bytes 2024-09-18T01:29:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2168624128 bytes 2024-09-18T01:29:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2174173184 bytes 2024-09-18T01:29:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2220365824 bytes 2024-09-18T01:29:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2235814912 bytes 2024-09-18T01:29:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2256307200 bytes 2024-09-18T01:29:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2262272000 bytes 2024-09-18T01:29:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2312825856 bytes 2024-09-18T01:29:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2313447424 bytes 2024-09-18T01:29:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2359872512 bytes 2024-09-18T01:29:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2413464576 bytes 2024-09-18T01:29:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2466839552 bytes 2024-09-18T01:29:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2516182016 bytes 2024-09-18T01:29:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2548867072 bytes 2024-09-18T01:29:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2582221824 bytes 2024-09-18T01:29:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2643212288 bytes 2024-09-18T01:29:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2747125760 bytes 2024-09-18T01:29:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2808326144 bytes 2024-09-18T01:29:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2854731776 bytes 2024-09-18T01:29:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2878134272 bytes 2024-09-18T01:29:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2952650752 bytes 2024-09-18T01:29:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2997007360 bytes 2024-09-18T01:29:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 3020025856 bytes 2024-09-18T01:29:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3045225472 bytes 2024-09-18T01:29:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3075506176 bytes 2024-09-18T01:29:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3088150528 bytes 2024-09-18T01:29:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3143464960 bytes 2024-09-18T01:29:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3190279168 bytes 2024-09-18T01:29:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3203660800 bytes 2024-09-18T01:29:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3237379072 bytes 2024-09-18T01:29:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3251254272 bytes 2024-09-18T01:29:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3279398912 bytes 2024-09-18T01:29:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3324334080 bytes 2024-09-18T01:29:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3405865984 bytes 2024-09-18T01:29:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3444951040 bytes 2024-09-18T01:29:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3516481536 bytes 2024-09-18T01:29:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3597621248 bytes 2024-09-18T01:29:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3658333184 bytes 2024-09-18T01:29:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3713496064 bytes 2024-09-18T01:29:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3748507648 bytes 2024-09-18T01:29:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3762967552 bytes 2024-09-18T01:29:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3787110400 bytes 2024-09-18T01:29:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3826680832 bytes 2024-09-18T01:29:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3904611328 bytes 2024-09-18T01:29:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3994931200 bytes 2024-09-18T01:29:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 4018927616 bytes 2024-09-18T01:29:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 4073040896 bytes 2024-09-18T01:29:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4127896576 bytes 2024-09-18T01:29:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4176936960 bytes 2024-09-18T01:29:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4199075840 bytes 2024-09-18T01:29:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4267485184 bytes 2024-09-18T01:29:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4308145152 bytes 2024-09-18T01:29:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4340495360 bytes 2024-09-18T01:29:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4384124928 bytes 2024-09-18T01:29:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4451893248 bytes 2024-09-18T01:29:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4492069888 bytes 2024-09-18T01:29:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4535906304 bytes 2024-09-18T01:29:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4564616192 bytes 2024-09-18T01:29:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:29:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:29:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:29:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:29:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_7 2024-09-18T01:29:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-18T01:29:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:29:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:29:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-18T01:29:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:29:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:29:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:29:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:29:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-18T01:29:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:29:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:29:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-18T01:29:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:29:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-18T01:29:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:29:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-18T01:29:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_7': 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 12b5cd36947e 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.12b5cd36947e 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:26 2024 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:26 2024 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:26 2024 2024-09-18T01:29:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:29:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:29:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 4597c48a-8662-495e-a897-2ee7f1408d91 2024-09-18T01:29:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:29:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:29:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-18T01:29:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:29:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:29:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:29:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:29:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:29:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 10263552 bytes 2024-09-18T01:29:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 48944128 bytes 2024-09-18T01:29:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 49553408 bytes 2024-09-18T01:29:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 84630528 bytes 2024-09-18T01:29:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 164418560 bytes 2024-09-18T01:29:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 175667200 bytes 2024-09-18T01:29:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 215269376 bytes 2024-09-18T01:29:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 221614080 bytes 2024-09-18T01:29:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 260092928 bytes 2024-09-18T01:29:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 297135104 bytes 2024-09-18T01:29:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 364991488 bytes 2024-09-18T01:29:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 449925120 bytes 2024-09-18T01:29:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 527437824 bytes 2024-09-18T01:29:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 553647104 bytes 2024-09-18T01:29:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 607782912 bytes 2024-09-18T01:29:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 714291200 bytes 2024-09-18T01:29:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 758985728 bytes 2024-09-18T01:29:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 788391936 bytes 2024-09-18T01:29:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 791810048 bytes 2024-09-18T01:29:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 804839424 bytes 2024-09-18T01:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 824868864 bytes 2024-09-18T01:29:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 833244160 bytes 2024-09-18T01:29:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 916376576 bytes 2024-09-18T01:29:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 970274816 bytes 2024-09-18T01:29:30.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1022450688 bytes 2024-09-18T01:29:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1082800128 bytes 2024-09-18T01:29:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1089345536 bytes 2024-09-18T01:29:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1103702016 bytes 2024-09-18T01:29:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1164296192 bytes 2024-09-18T01:29:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1164918784 bytes 2024-09-18T01:29:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1189447680 bytes 2024-09-18T01:29:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1290876928 bytes 2024-09-18T01:29:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1350439936 bytes 2024-09-18T01:29:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1387461632 bytes 2024-09-18T01:29:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1432963072 bytes 2024-09-18T01:29:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1460493312 bytes 2024-09-18T01:29:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1470696448 bytes 2024-09-18T01:29:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1486077952 bytes 2024-09-18T01:29:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1499409408 bytes 2024-09-18T01:29:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1525883904 bytes 2024-09-18T01:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1559190528 bytes 2024-09-18T01:29:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1581869056 bytes 2024-09-18T01:29:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1617375232 bytes 2024-09-18T01:29:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1644649472 bytes 2024-09-18T01:29:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1668795392 bytes 2024-09-18T01:29:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1702047744 bytes 2024-09-18T01:29:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1730883584 bytes 2024-09-18T01:29:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1788044288 bytes 2024-09-18T01:29:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1813215232 bytes 2024-09-18T01:29:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1827685376 bytes 2024-09-18T01:29:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1871546368 bytes 2024-09-18T01:29:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1872806912 bytes 2024-09-18T01:29:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 1955399680 bytes 2024-09-18T01:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 1976896512 bytes 2024-09-18T01:29:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2011745280 bytes 2024-09-18T01:29:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2037253120 bytes 2024-09-18T01:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2068931584 bytes 2024-09-18T01:29:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:29:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:29:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:29:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:29:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_8 2024-09-18T01:29:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-18T01:29:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:29:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:29:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:29:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:29:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-18T01:29:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:29:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_8': 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 12c11c5e1411 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.12c11c5e1411 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:29:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:29:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:33 2024 2024-09-18T01:29:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:33 2024 2024-09-18T01:29:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:33 2024 2024-09-18T01:29:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:29:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:29:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: b4e5b9d2-efbb-4d6e-b309-ef7dc24ba029 2024-09-18T01:29:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:29:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:29:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-18T01:29:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:29:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:29:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:29:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:29:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:29:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 94082048 bytes 2024-09-18T01:29:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 133972992 bytes 2024-09-18T01:29:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 227219456 bytes 2024-09-18T01:29:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 272604160 bytes 2024-09-18T01:29:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 310467584 bytes 2024-09-18T01:29:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 348537856 bytes 2024-09-18T01:29:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 441108480 bytes 2024-09-18T01:29:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 452316160 bytes 2024-09-18T01:29:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 517903360 bytes 2024-09-18T01:29:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 534669312 bytes 2024-09-18T01:29:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 663706624 bytes 2024-09-18T01:29:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 722108416 bytes 2024-09-18T01:29:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 751128576 bytes 2024-09-18T01:29:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 821650432 bytes 2024-09-18T01:29:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 885114880 bytes 2024-09-18T01:29:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 908226560 bytes 2024-09-18T01:29:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 954786816 bytes 2024-09-18T01:29:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 1005705216 bytes 2024-09-18T01:29:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 1035491328 bytes 2024-09-18T01:29:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1148845056 bytes 2024-09-18T01:29:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1171602432 bytes 2024-09-18T01:29:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1197085696 bytes 2024-09-18T01:29:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1267002368 bytes 2024-09-18T01:29:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1295655936 bytes 2024-09-18T01:29:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1335936000 bytes 2024-09-18T01:29:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1403922432 bytes 2024-09-18T01:29:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1413102592 bytes 2024-09-18T01:29:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1451183104 bytes 2024-09-18T01:29:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1487193088 bytes 2024-09-18T01:29:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1540495360 bytes 2024-09-18T01:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1586430976 bytes 2024-09-18T01:29:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1640595456 bytes 2024-09-18T01:29:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1720701952 bytes 2024-09-18T01:29:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1735217152 bytes 2024-09-18T01:29:39.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1793303552 bytes 2024-09-18T01:29:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1855106048 bytes 2024-09-18T01:29:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1877553152 bytes 2024-09-18T01:29:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1926909952 bytes 2024-09-18T01:29:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 2030192640 bytes 2024-09-18T01:29:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 2030819328 bytes 2024-09-18T01:29:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 2065147904 bytes 2024-09-18T01:29:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2142406656 bytes 2024-09-18T01:29:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2179868672 bytes 2024-09-18T01:29:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2218148864 bytes 2024-09-18T01:29:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2236278784 bytes 2024-09-18T01:29:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2297006080 bytes 2024-09-18T01:29:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2359587840 bytes 2024-09-18T01:29:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2439005184 bytes 2024-09-18T01:29:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2439604224 bytes 2024-09-18T01:29:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2501256192 bytes 2024-09-18T01:29:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2558429184 bytes 2024-09-18T01:29:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2606052352 bytes 2024-09-18T01:29:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2606669824 bytes 2024-09-18T01:29:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2616181760 bytes 2024-09-18T01:29:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2671645696 bytes 2024-09-18T01:29:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2775388160 bytes 2024-09-18T01:29:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2829426688 bytes 2024-09-18T01:29:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2858616832 bytes 2024-09-18T01:29:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2900101120 bytes 2024-09-18T01:29:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2901714944 bytes 2024-09-18T01:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2927856640 bytes 2024-09-18T01:29:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2940426240 bytes 2024-09-18T01:29:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2996503552 bytes 2024-09-18T01:29:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 3079077888 bytes 2024-09-18T01:29:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 3087865856 bytes 2024-09-18T01:29:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 3109694464 bytes 2024-09-18T01:29:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3207929856 bytes 2024-09-18T01:29:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3253216256 bytes 2024-09-18T01:29:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3321920512 bytes 2024-09-18T01:29:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3325134848 bytes 2024-09-18T01:29:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3361696768 bytes 2024-09-18T01:29:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3413258240 bytes 2024-09-18T01:29:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3504491520 bytes 2024-09-18T01:29:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3555768320 bytes 2024-09-18T01:29:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3619322880 bytes 2024-09-18T01:29:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3702365184 bytes 2024-09-18T01:29:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:29:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:29:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_9 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:29:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:29:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-18T01:29:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:29:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_9': 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 12cdc56135e5 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.12cdc56135e5 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:29:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:43 2024 2024-09-18T01:29:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:43 2024 2024-09-18T01:29:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:43 2024 2024-09-18T01:29:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:29:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:29:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 68d70003-8fef-4a72-b907-b819e314282e 2024-09-18T01:29:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:29:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:29:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-18T01:29:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:29:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:29:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=7 2024-09-18T01:29:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:29:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:29:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 29074432 bytes 2024-09-18T01:29:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 47594496 bytes 2024-09-18T01:29:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 115898368 bytes 2024-09-18T01:29:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 168768512 bytes 2024-09-18T01:29:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 189254656 bytes 2024-09-18T01:29:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 247036928 bytes 2024-09-18T01:29:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 283792384 bytes 2024-09-18T01:29:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 309537792 bytes 2024-09-18T01:29:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 388127744 bytes 2024-09-18T01:29:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 431673344 bytes 2024-09-18T01:29:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 499706880 bytes 2024-09-18T01:29:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 541127680 bytes 2024-09-18T01:29:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 624505856 bytes 2024-09-18T01:29:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 770433024 bytes 2024-09-18T01:29:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 817525760 bytes 2024-09-18T01:29:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 919579648 bytes 2024-09-18T01:29:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 937995264 bytes 2024-09-18T01:29:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 993986560 bytes 2024-09-18T01:29:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 1048110080 bytes 2024-09-18T01:29:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1126788096 bytes 2024-09-18T01:29:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1166546944 bytes 2024-09-18T01:29:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1186442240 bytes 2024-09-18T01:29:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1200978944 bytes 2024-09-18T01:29:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1256603648 bytes 2024-09-18T01:29:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1309961216 bytes 2024-09-18T01:29:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1350006784 bytes 2024-09-18T01:29:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1390088192 bytes 2024-09-18T01:29:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1437188096 bytes 2024-09-18T01:29:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1459788800 bytes 2024-09-18T01:29:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1460392960 bytes 2024-09-18T01:29:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1514491904 bytes 2024-09-18T01:29:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1544265728 bytes 2024-09-18T01:29:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1592950784 bytes 2024-09-18T01:29:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1705124864 bytes 2024-09-18T01:29:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1791801344 bytes 2024-09-18T01:29:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1839804416 bytes 2024-09-18T01:29:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1876494336 bytes 2024-09-18T01:29:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1899361280 bytes 2024-09-18T01:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1955701760 bytes 2024-09-18T01:29:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1981860864 bytes 2024-09-18T01:29:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 2074795008 bytes 2024-09-18T01:29:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2100224000 bytes 2024-09-18T01:29:49.715 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3892955734878626) > weight (0.25) 2024-09-18T01:29:49.715 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3611288813662794) > weight (0.25) 2024-09-18T01:29:49.715 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-18T01:29:49.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-18T01:29:49.716 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-18T01:29:49.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:29:49.715+0000 7ff68561d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 36772) UID: 1000 2024-09-18T01:29:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2206124032 bytes 2024-09-18T01:29:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2281883648 bytes 2024-09-18T01:29:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2331089920 bytes 2024-09-18T01:29:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2375028736 bytes 2024-09-18T01:29:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2430280704 bytes 2024-09-18T01:29:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2444621824 bytes 2024-09-18T01:29:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2493807616 bytes 2024-09-18T01:29:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2499266560 bytes 2024-09-18T01:29:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2518011904 bytes 2024-09-18T01:29:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2567280640 bytes 2024-09-18T01:29:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2624694272 bytes 2024-09-18T01:29:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2690689024 bytes 2024-09-18T01:29:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2722088960 bytes 2024-09-18T01:29:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2742207488 bytes 2024-09-18T01:29:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2777270272 bytes 2024-09-18T01:29:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2800872448 bytes 2024-09-18T01:29:51.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2835765248 bytes 2024-09-18T01:29:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2884178944 bytes 2024-09-18T01:29:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_10 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:29:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:29:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:29:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-18T01:29:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:29:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:29:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-18T01:29:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:29:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-18T01:29:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:29:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-18T01:29:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_10': 2024-09-18T01:29:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:29:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:29:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 12d9dc44733a 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.12d9dc44733a 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:51 2024 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:51 2024 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:51 2024 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: c330e23e-03a4-45ae-9fa0-73ca0a1ac911 2024-09-18T01:29:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:29:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:29:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-18T01:29:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:29:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:29:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=7 2024-09-18T01:29:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:29:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:29:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 69468160 bytes 2024-09-18T01:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 91881472 bytes 2024-09-18T01:29:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 168553472 bytes 2024-09-18T01:29:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 195826688 bytes 2024-09-18T01:29:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 217165824 bytes 2024-09-18T01:29:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 265147392 bytes 2024-09-18T01:29:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 304243712 bytes 2024-09-18T01:29:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 352636928 bytes 2024-09-18T01:29:53.829 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6516329401476111) > weight (0.25) 2024-09-18T01:29:53.829 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.36975230094293576) > weight (0.25) 2024-09-18T01:29:53.829 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3018960073806788) > weight (0.25) 2024-09-18T01:29:53.829 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-18T01:29:53.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-18T01:29:53.829 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-18T01:29:53.829 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-18T01:29:53.830 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:29:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 378747904 bytes 2024-09-18T01:29:53.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:29:53.828+0000 7f11c1292640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 36880) UID: 1000 2024-09-18T01:29:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:29:53.832+0000 7fadd68ba640 -1 reset not still connected to 0x55d6a3e33930 2024-09-18T01:29:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:29:53.832+0000 7fadd68ba640 -1 reset not still connected to 0x55d6a4118820 2024-09-18T01:29:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 379388928 bytes 2024-09-18T01:29:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 439818240 bytes 2024-09-18T01:29:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 440434688 bytes 2024-09-18T01:29:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 465415168 bytes 2024-09-18T01:29:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 484318208 bytes 2024-09-18T01:29:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 506973184 bytes 2024-09-18T01:29:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 550487040 bytes 2024-09-18T01:29:54.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 551090176 bytes 2024-09-18T01:29:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 563639296 bytes 2024-09-18T01:29:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 580938752 bytes 2024-09-18T01:29:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 611153920 bytes 2024-09-18T01:29:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 667702272 bytes 2024-09-18T01:29:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 682743808 bytes 2024-09-18T01:29:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 737566720 bytes 2024-09-18T01:29:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 778431488 bytes 2024-09-18T01:29:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 852358144 bytes 2024-09-18T01:29:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 886727680 bytes 2024-09-18T01:29:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 979126272 bytes 2024-09-18T01:29:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1042821120 bytes 2024-09-18T01:29:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1085342720 bytes 2024-09-18T01:29:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1159954432 bytes 2024-09-18T01:29:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1205890048 bytes 2024-09-18T01:29:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1236419584 bytes 2024-09-18T01:29:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1259317248 bytes 2024-09-18T01:29:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1305033728 bytes 2024-09-18T01:29:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1335755776 bytes 2024-09-18T01:29:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1395723264 bytes 2024-09-18T01:29:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1454797824 bytes 2024-09-18T01:29:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1475761152 bytes 2024-09-18T01:29:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1509804032 bytes 2024-09-18T01:29:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1516463104 bytes 2024-09-18T01:29:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1551804416 bytes 2024-09-18T01:29:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1570084864 bytes 2024-09-18T01:29:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1649582080 bytes 2024-09-18T01:29:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1694220288 bytes 2024-09-18T01:29:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1701720064 bytes 2024-09-18T01:29:57.716 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-18T01:29:57.716 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:29:57.716 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-18T01:29:57.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-18T01:29:57.717 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-18T01:29:57.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-18T01:29:57.720 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-09-18T01:29:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1722269696 bytes 2024-09-18T01:29:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1774268416 bytes 2024-09-18T01:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1774887936 bytes 2024-09-18T01:29:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1855629312 bytes 2024-09-18T01:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1950143488 bytes 2024-09-18T01:29:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2004608000 bytes 2024-09-18T01:29:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2041744384 bytes 2024-09-18T01:29:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2135647232 bytes 2024-09-18T01:29:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2150141952 bytes 2024-09-18T01:29:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2210066432 bytes 2024-09-18T01:29:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2253193216 bytes 2024-09-18T01:29:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2266081280 bytes 2024-09-18T01:29:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2300576768 bytes 2024-09-18T01:29:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2337276928 bytes 2024-09-18T01:29:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2353263616 bytes 2024-09-18T01:29:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2364101632 bytes 2024-09-18T01:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2396113920 bytes 2024-09-18T01:29:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2442471424 bytes 2024-09-18T01:29:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:29:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:29:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:29:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_11 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-18T01:29:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2024-09-18T01:29:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:29:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:29:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:29:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-18T01:29:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:29:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:29:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:29:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:29:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-18T01:29:59.932 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-18T01:29:59.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-18T01:29:59.932 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-18T01:29:59.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-18T01:29:59.935 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-18T01:30:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:30:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-18T01:30:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_11': 2024-09-18T01:30:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:30:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 12ea1325c9c1 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.12ea1325c9c1 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:59 2024 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:59 2024 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:59 2024 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:30:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: f4d6f564-c302-4459-b459-852582eb1804 2024-09-18T01:30:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:30:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:30:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-18T01:30:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:30:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:30:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:30:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:30:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:30:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 32158720 bytes 2024-09-18T01:30:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 103460864 bytes 2024-09-18T01:30:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 163959808 bytes 2024-09-18T01:30:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 200514560 bytes 2024-09-18T01:30:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 237449216 bytes 2024-09-18T01:30:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 301623296 bytes 2024-09-18T01:30:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 348172288 bytes 2024-09-18T01:30:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 369473536 bytes 2024-09-18T01:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 380001280 bytes 2024-09-18T01:30:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 418582528 bytes 2024-09-18T01:30:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 469376000 bytes 2024-09-18T01:30:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 480764928 bytes 2024-09-18T01:30:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 559102976 bytes 2024-09-18T01:30:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 598305792 bytes 2024-09-18T01:30:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 656025600 bytes 2024-09-18T01:30:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 673380352 bytes 2024-09-18T01:30:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 697507840 bytes 2024-09-18T01:30:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 774944768 bytes 2024-09-18T01:30:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 813059072 bytes 2024-09-18T01:30:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 837507072 bytes 2024-09-18T01:30:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 862189568 bytes 2024-09-18T01:30:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 885624832 bytes 2024-09-18T01:30:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 926161920 bytes 2024-09-18T01:30:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 951050240 bytes 2024-09-18T01:30:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 953349120 bytes 2024-09-18T01:30:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 998912000 bytes 2024-09-18T01:30:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1047799808 bytes 2024-09-18T01:30:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1077868544 bytes 2024-09-18T01:30:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1102656512 bytes 2024-09-18T01:30:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1143171072 bytes 2024-09-18T01:30:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1175378944 bytes 2024-09-18T01:30:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1240488960 bytes 2024-09-18T01:30:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1285755904 bytes 2024-09-18T01:30:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1303799808 bytes 2024-09-18T01:30:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1338504192 bytes 2024-09-18T01:30:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1384695808 bytes 2024-09-18T01:30:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1429619712 bytes 2024-09-18T01:30:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1482295296 bytes 2024-09-18T01:30:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1515848704 bytes 2024-09-18T01:30:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1538566144 bytes 2024-09-18T01:30:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1610131456 bytes 2024-09-18T01:30:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1687832576 bytes 2024-09-18T01:30:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1776017408 bytes 2024-09-18T01:30:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:30:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_12 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-18T01:30:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:30:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_12': 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 12ff8bf84bf3 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.12ff8bf84bf3 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:30:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:30:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:30:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:05 2024 2024-09-18T01:30:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:05 2024 2024-09-18T01:30:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:05 2024 2024-09-18T01:30:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:30:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:30:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: d652dfd4-0db3-409b-a60f-853dfb4b8bbf 2024-09-18T01:30:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:30:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:30:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-18T01:30:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:30:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:30:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:30:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:30:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:30:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 31835136 bytes 2024-09-18T01:30:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 32424960 bytes 2024-09-18T01:30:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 87628800 bytes 2024-09-18T01:30:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 118839296 bytes 2024-09-18T01:30:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 164337664 bytes 2024-09-18T01:30:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 192963584 bytes 2024-09-18T01:30:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 205796352 bytes 2024-09-18T01:30:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 211725312 bytes 2024-09-18T01:30:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 246292480 bytes 2024-09-18T01:30:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 389265408 bytes 2024-09-18T01:30:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 415614976 bytes 2024-09-18T01:30:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 450143232 bytes 2024-09-18T01:30:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 493012992 bytes 2024-09-18T01:30:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 533808128 bytes 2024-09-18T01:30:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 566528000 bytes 2024-09-18T01:30:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 583922688 bytes 2024-09-18T01:30:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 610393088 bytes 2024-09-18T01:30:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 628240384 bytes 2024-09-18T01:30:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 671918080 bytes 2024-09-18T01:30:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 695001088 bytes 2024-09-18T01:30:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 727221248 bytes 2024-09-18T01:30:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 733023232 bytes 2024-09-18T01:30:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 768757760 bytes 2024-09-18T01:30:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 806035456 bytes 2024-09-18T01:30:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 854393856 bytes 2024-09-18T01:30:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 891218944 bytes 2024-09-18T01:30:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 978542592 bytes 2024-09-18T01:30:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1063495680 bytes 2024-09-18T01:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1114178560 bytes 2024-09-18T01:30:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1150258176 bytes 2024-09-18T01:30:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1172362240 bytes 2024-09-18T01:30:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1211602944 bytes 2024-09-18T01:30:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1226984448 bytes 2024-09-18T01:30:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1230167040 bytes 2024-09-18T01:30:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1244431360 bytes 2024-09-18T01:30:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1273449472 bytes 2024-09-18T01:30:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1312780288 bytes 2024-09-18T01:30:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1386553344 bytes 2024-09-18T01:30:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1432819712 bytes 2024-09-18T01:30:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1446969344 bytes 2024-09-18T01:30:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1457351680 bytes 2024-09-18T01:30:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1511610368 bytes 2024-09-18T01:30:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1550997504 bytes 2024-09-18T01:30:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1607211008 bytes 2024-09-18T01:30:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1677805568 bytes 2024-09-18T01:30:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1695283200 bytes 2024-09-18T01:30:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:30:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_13 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:30:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-18T01:30:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2024-09-18T01:30:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:30:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:30:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-18T01:30:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:30:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-18T01:30:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:30:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-18T01:30:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_13': 2024-09-18T01:30:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:30:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:30:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 130b66db7e24 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.130b66db7e24 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:11 2024 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:11 2024 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:11 2024 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 0b20a1f2-779a-42c1-bde9-cc45daa1eb9b 2024-09-18T01:30:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:30:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:30:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-18T01:30:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:30:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:30:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=11 2024-09-18T01:30:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:30:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:30:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 36484096 bytes 2024-09-18T01:30:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 52679680 bytes 2024-09-18T01:30:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 126262272 bytes 2024-09-18T01:30:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 180210688 bytes 2024-09-18T01:30:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 184815616 bytes 2024-09-18T01:30:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 227056640 bytes 2024-09-18T01:30:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 227663872 bytes 2024-09-18T01:30:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 345751552 bytes 2024-09-18T01:30:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 364044288 bytes 2024-09-18T01:30:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 375571456 bytes 2024-09-18T01:30:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 376198144 bytes 2024-09-18T01:30:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 392852480 bytes 2024-09-18T01:30:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 429560832 bytes 2024-09-18T01:30:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 463921152 bytes 2024-09-18T01:30:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 551064576 bytes 2024-09-18T01:30:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 586098688 bytes 2024-09-18T01:30:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 624752640 bytes 2024-09-18T01:30:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 669101056 bytes 2024-09-18T01:30:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 690541568 bytes 2024-09-18T01:30:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 747852800 bytes 2024-09-18T01:30:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 771909632 bytes 2024-09-18T01:30:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 794785792 bytes 2024-09-18T01:30:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 833846272 bytes 2024-09-18T01:30:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 881994752 bytes 2024-09-18T01:30:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 911360000 bytes 2024-09-18T01:30:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 918104064 bytes 2024-09-18T01:30:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 918731776 bytes 2024-09-18T01:30:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 926828544 bytes 2024-09-18T01:30:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 956821504 bytes 2024-09-18T01:30:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1001216000 bytes 2024-09-18T01:30:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1056807936 bytes 2024-09-18T01:30:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1105516544 bytes 2024-09-18T01:30:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1151039488 bytes 2024-09-18T01:30:16.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1188064256 bytes 2024-09-18T01:30:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1219494912 bytes 2024-09-18T01:30:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1341845504 bytes 2024-09-18T01:30:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1383598080 bytes 2024-09-18T01:30:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1440701440 bytes 2024-09-18T01:30:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1477451776 bytes 2024-09-18T01:30:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1491807232 bytes 2024-09-18T01:30:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1557129216 bytes 2024-09-18T01:30:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1599248384 bytes 2024-09-18T01:30:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1625421824 bytes 2024-09-18T01:30:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1654757376 bytes 2024-09-18T01:30:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1669702656 bytes 2024-09-18T01:30:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1713876992 bytes 2024-09-18T01:30:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1743390720 bytes 2024-09-18T01:30:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1795353600 bytes 2024-09-18T01:30:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1886891008 bytes 2024-09-18T01:30:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1942326272 bytes 2024-09-18T01:30:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1976570880 bytes 2024-09-18T01:30:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2035018752 bytes 2024-09-18T01:30:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2056679424 bytes 2024-09-18T01:30:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2063710208 bytes 2024-09-18T01:30:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2126445568 bytes 2024-09-18T01:30:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2190212096 bytes 2024-09-18T01:30:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2230224896 bytes 2024-09-18T01:30:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2278200320 bytes 2024-09-18T01:30:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2361265152 bytes 2024-09-18T01:30:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2410170368 bytes 2024-09-18T01:30:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2468931584 bytes 2024-09-18T01:30:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2517400576 bytes 2024-09-18T01:30:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2629076992 bytes 2024-09-18T01:30:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2649469952 bytes 2024-09-18T01:30:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2723135488 bytes 2024-09-18T01:30:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2723730432 bytes 2024-09-18T01:30:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2730480640 bytes 2024-09-18T01:30:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2750052352 bytes 2024-09-18T01:30:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2797852672 bytes 2024-09-18T01:30:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2821066752 bytes 2024-09-18T01:30:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2828954624 bytes 2024-09-18T01:30:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2858443776 bytes 2024-09-18T01:30:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2906218496 bytes 2024-09-18T01:30:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 2973678592 bytes 2024-09-18T01:30:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 2978625536 bytes 2024-09-18T01:30:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3041697792 bytes 2024-09-18T01:30:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3064371200 bytes 2024-09-18T01:30:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3155419136 bytes 2024-09-18T01:30:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3212611584 bytes 2024-09-18T01:30:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3267917824 bytes 2024-09-18T01:30:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3313318912 bytes 2024-09-18T01:30:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3353791488 bytes 2024-09-18T01:30:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3354388480 bytes 2024-09-18T01:30:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3376750592 bytes 2024-09-18T01:30:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3391943680 bytes 2024-09-18T01:30:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3415279616 bytes 2024-09-18T01:30:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3439834112 bytes 2024-09-18T01:30:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3470696448 bytes 2024-09-18T01:30:22.935 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:30:22.935+0000 7f678fb95640 -1 reset not still connected to 0x56465ebb2f70 2024-09-18T01:30:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3483300864 bytes 2024-09-18T01:30:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3544043520 bytes 2024-09-18T01:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3546478592 bytes 2024-09-18T01:30:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3556082688 bytes 2024-09-18T01:30:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 3623170048 bytes 2024-09-18T01:30:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 3708484608 bytes 2024-09-18T01:30:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 3737893888 bytes 2024-09-18T01:30:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 3775304704 bytes 2024-09-18T01:30:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_14 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:30:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-18T01:30:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:30:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-18T01:30:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_14': 2024-09-18T01:30:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:30:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:30:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:30:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 131771a5ad13 2024-09-18T01:30:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.131771a5ad13 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:23 2024 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:23 2024 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:23 2024 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3c45b3f0-662d-4182-8d89-837d0b48bc1a 2024-09-18T01:30:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:30:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:30:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-18T01:30:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:30:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:30:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=10 2024-09-18T01:30:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:30:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:30:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 61648896 bytes 2024-09-18T01:30:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 123826176 bytes 2024-09-18T01:30:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 153095168 bytes 2024-09-18T01:30:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 200354816 bytes 2024-09-18T01:30:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 219148288 bytes 2024-09-18T01:30:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 252279808 bytes 2024-09-18T01:30:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 352653312 bytes 2024-09-18T01:30:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 376048640 bytes 2024-09-18T01:30:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 397681664 bytes 2024-09-18T01:30:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 441600000 bytes 2024-09-18T01:30:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 476047360 bytes 2024-09-18T01:30:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 476658688 bytes 2024-09-18T01:30:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 498629632 bytes 2024-09-18T01:30:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 620862464 bytes 2024-09-18T01:30:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 638946304 bytes 2024-09-18T01:30:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 639586304 bytes 2024-09-18T01:30:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 701501440 bytes 2024-09-18T01:30:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 780335104 bytes 2024-09-18T01:30:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 827651072 bytes 2024-09-18T01:30:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 885524480 bytes 2024-09-18T01:30:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 916255744 bytes 2024-09-18T01:30:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 978383872 bytes 2024-09-18T01:30:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1011076096 bytes 2024-09-18T01:30:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1064968192 bytes 2024-09-18T01:30:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1074723840 bytes 2024-09-18T01:30:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1078958080 bytes 2024-09-18T01:30:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1088350208 bytes 2024-09-18T01:30:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1105727488 bytes 2024-09-18T01:30:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1194564608 bytes 2024-09-18T01:30:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1215364096 bytes 2024-09-18T01:30:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1293119488 bytes 2024-09-18T01:30:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1317287936 bytes 2024-09-18T01:30:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1393632256 bytes 2024-09-18T01:30:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1442484224 bytes 2024-09-18T01:30:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1444156416 bytes 2024-09-18T01:30:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1528901632 bytes 2024-09-18T01:30:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1532508160 bytes 2024-09-18T01:30:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1584019456 bytes 2024-09-18T01:30:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1692049408 bytes 2024-09-18T01:30:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1775792128 bytes 2024-09-18T01:30:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1861577728 bytes 2024-09-18T01:30:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1903843328 bytes 2024-09-18T01:30:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1923667968 bytes 2024-09-18T01:30:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1943139328 bytes 2024-09-18T01:30:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1943747584 bytes 2024-09-18T01:30:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2009612288 bytes 2024-09-18T01:30:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2034354176 bytes 2024-09-18T01:30:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2070123520 bytes 2024-09-18T01:30:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2087930880 bytes 2024-09-18T01:30:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2114423808 bytes 2024-09-18T01:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2144035840 bytes 2024-09-18T01:30:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2182493184 bytes 2024-09-18T01:30:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2227426304 bytes 2024-09-18T01:30:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2228021248 bytes 2024-09-18T01:30:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2271191040 bytes 2024-09-18T01:30:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2313495552 bytes 2024-09-18T01:30:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2336334848 bytes 2024-09-18T01:30:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2383345664 bytes 2024-09-18T01:30:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2424912896 bytes 2024-09-18T01:30:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2477257728 bytes 2024-09-18T01:30:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2509906944 bytes 2024-09-18T01:30:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2533626880 bytes 2024-09-18T01:30:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2535940096 bytes 2024-09-18T01:30:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2564517888 bytes 2024-09-18T01:30:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2565093376 bytes 2024-09-18T01:30:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2589101056 bytes 2024-09-18T01:30:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2642395136 bytes 2024-09-18T01:30:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2655581184 bytes 2024-09-18T01:30:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2681765888 bytes 2024-09-18T01:30:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2723492864 bytes 2024-09-18T01:30:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2741076992 bytes 2024-09-18T01:30:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2857237504 bytes 2024-09-18T01:30:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2892769280 bytes 2024-09-18T01:30:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 2913894400 bytes 2024-09-18T01:30:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 2958546944 bytes 2024-09-18T01:30:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 2996348928 bytes 2024-09-18T01:30:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3052278784 bytes 2024-09-18T01:30:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3121690624 bytes 2024-09-18T01:30:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3203687424 bytes 2024-09-18T01:30:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3240931328 bytes 2024-09-18T01:30:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3272267776 bytes 2024-09-18T01:30:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3321621504 bytes 2024-09-18T01:30:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3380436992 bytes 2024-09-18T01:30:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3421458432 bytes 2024-09-18T01:30:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:30:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:30:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:30:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_15 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:30:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-18T01:30:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2024-09-18T01:30:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:30:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:30:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-18T01:30:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:30:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-18T01:30:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:30:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-18T01:30:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_15': 2024-09-18T01:30:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:30:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:30:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 132386941f35 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.132386941f35 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:34 2024 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:34 2024 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:34 2024 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 15b04f61-470a-4fa7-811f-b4270495f8f7 2024-09-18T01:30:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:30:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:30:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-18T01:30:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:30:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:30:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:30:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:30:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:30:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 85680128 bytes 2024-09-18T01:30:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 109811712 bytes 2024-09-18T01:30:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 124654592 bytes 2024-09-18T01:30:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 125284352 bytes 2024-09-18T01:30:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 167323648 bytes 2024-09-18T01:30:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 250692608 bytes 2024-09-18T01:30:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 313713664 bytes 2024-09-18T01:30:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 354613248 bytes 2024-09-18T01:30:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 355229696 bytes 2024-09-18T01:30:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 403712000 bytes 2024-09-18T01:30:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 470468608 bytes 2024-09-18T01:30:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 530672640 bytes 2024-09-18T01:30:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 537125888 bytes 2024-09-18T01:30:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 583898112 bytes 2024-09-18T01:30:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 653965312 bytes 2024-09-18T01:30:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 718901248 bytes 2024-09-18T01:30:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 755113984 bytes 2024-09-18T01:30:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 791369728 bytes 2024-09-18T01:30:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 866506752 bytes 2024-09-18T01:30:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 965107712 bytes 2024-09-18T01:30:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1041724416 bytes 2024-09-18T01:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1077103616 bytes 2024-09-18T01:30:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1077687296 bytes 2024-09-18T01:30:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1103273984 bytes 2024-09-18T01:30:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1103882240 bytes 2024-09-18T01:30:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1142036480 bytes 2024-09-18T01:30:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1180929024 bytes 2024-09-18T01:30:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1199881216 bytes 2024-09-18T01:30:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1278649344 bytes 2024-09-18T01:30:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1302421504 bytes 2024-09-18T01:30:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1345468416 bytes 2024-09-18T01:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1369172992 bytes 2024-09-18T01:30:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1401013248 bytes 2024-09-18T01:30:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1483003904 bytes 2024-09-18T01:30:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1526450176 bytes 2024-09-18T01:30:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1531931648 bytes 2024-09-18T01:30:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1545861120 bytes 2024-09-18T01:30:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1618718720 bytes 2024-09-18T01:30:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1716018176 bytes 2024-09-18T01:30:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1735139328 bytes 2024-09-18T01:30:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1818700800 bytes 2024-09-18T01:30:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1846806528 bytes 2024-09-18T01:30:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1924483072 bytes 2024-09-18T01:30:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1960763392 bytes 2024-09-18T01:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1961395200 bytes 2024-09-18T01:30:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2005321728 bytes 2024-09-18T01:30:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2036959232 bytes 2024-09-18T01:30:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2052719616 bytes 2024-09-18T01:30:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2121265152 bytes 2024-09-18T01:30:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2144077824 bytes 2024-09-18T01:30:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2167856128 bytes 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_16 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:30:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:30:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:30:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:30:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2024-09-18T01:30:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:30:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:30:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-18T01:30:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:30:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-18T01:30:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:30:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-18T01:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_16': 2024-09-18T01:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 132ff1ed72bf 2024-09-18T01:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.132ff1ed72bf 2024-09-18T01:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:41 2024 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:41 2024 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:41 2024 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 0a9c0c62-9c63-4428-abf9-46d978cda72b 2024-09-18T01:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:30:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:30:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-18T01:30:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:30:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:30:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:30:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:30:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:30:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 3971072 bytes 2024-09-18T01:30:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 48782336 bytes 2024-09-18T01:30:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 97040384 bytes 2024-09-18T01:30:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 167016448 bytes 2024-09-18T01:30:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 171145216 bytes 2024-09-18T01:30:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 230625280 bytes 2024-09-18T01:30:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 253296640 bytes 2024-09-18T01:30:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 312842240 bytes 2024-09-18T01:30:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 353145856 bytes 2024-09-18T01:30:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 369580032 bytes 2024-09-18T01:30:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 393292800 bytes 2024-09-18T01:30:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 498974720 bytes 2024-09-18T01:30:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 517245952 bytes 2024-09-18T01:30:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 544107520 bytes 2024-09-18T01:30:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 573727744 bytes 2024-09-18T01:30:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 589534208 bytes 2024-09-18T01:30:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 637741056 bytes 2024-09-18T01:30:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 710483968 bytes 2024-09-18T01:30:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 720226304 bytes 2024-09-18T01:30:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 784168960 bytes 2024-09-18T01:30:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 850024448 bytes 2024-09-18T01:30:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 870637568 bytes 2024-09-18T01:30:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 911240192 bytes 2024-09-18T01:30:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 929056768 bytes 2024-09-18T01:30:45.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 945902592 bytes 2024-09-18T01:30:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 997147648 bytes 2024-09-18T01:30:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1034823680 bytes 2024-09-18T01:30:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1070346240 bytes 2024-09-18T01:30:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1130567680 bytes 2024-09-18T01:30:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1157849088 bytes 2024-09-18T01:30:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1164190720 bytes 2024-09-18T01:30:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1200054272 bytes 2024-09-18T01:30:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1267283968 bytes 2024-09-18T01:30:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1295177728 bytes 2024-09-18T01:30:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1382603776 bytes 2024-09-18T01:30:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1448450048 bytes 2024-09-18T01:30:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1483953152 bytes 2024-09-18T01:30:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1493832704 bytes 2024-09-18T01:30:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1525536768 bytes 2024-09-18T01:30:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1614711808 bytes 2024-09-18T01:30:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1654355968 bytes 2024-09-18T01:30:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1682485248 bytes 2024-09-18T01:30:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1698728960 bytes 2024-09-18T01:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_17 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:30:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:30:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-18T01:30:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2024-09-18T01:30:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:30:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:30:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-18T01:30:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:30:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-18T01:30:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:30:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_17': 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 133bed58e71a 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.133bed58e71a 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:47 2024 2024-09-18T01:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:47 2024 2024-09-18T01:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:47 2024 2024-09-18T01:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 2af45ec2-1273-4f89-9d4a-659567f6c7cc 2024-09-18T01:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-18T01:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:30:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=7 2024-09-18T01:30:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:30:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:30:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 92346368 bytes 2024-09-18T01:30:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 124200960 bytes 2024-09-18T01:30:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 169465856 bytes 2024-09-18T01:30:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 213225472 bytes 2024-09-18T01:30:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 294049792 bytes 2024-09-18T01:30:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 336498688 bytes 2024-09-18T01:30:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 406505472 bytes 2024-09-18T01:30:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 451961856 bytes 2024-09-18T01:30:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 497280000 bytes 2024-09-18T01:30:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 499356672 bytes 2024-09-18T01:30:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 516766720 bytes 2024-09-18T01:30:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 602688512 bytes 2024-09-18T01:30:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 647952384 bytes 2024-09-18T01:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 688029696 bytes 2024-09-18T01:30:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 736032768 bytes 2024-09-18T01:30:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 769559552 bytes 2024-09-18T01:30:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 791124992 bytes 2024-09-18T01:30:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 822140928 bytes 2024-09-18T01:30:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 840755200 bytes 2024-09-18T01:30:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 879624192 bytes 2024-09-18T01:30:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 901981184 bytes 2024-09-18T01:30:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 944048128 bytes 2024-09-18T01:30:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 989821952 bytes 2024-09-18T01:30:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1006581760 bytes 2024-09-18T01:30:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1052730368 bytes 2024-09-18T01:30:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1185468416 bytes 2024-09-18T01:30:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1279716352 bytes 2024-09-18T01:30:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1329271808 bytes 2024-09-18T01:30:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1363297280 bytes 2024-09-18T01:30:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1375659008 bytes 2024-09-18T01:30:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1403111424 bytes 2024-09-18T01:30:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1459788800 bytes 2024-09-18T01:30:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1475060736 bytes 2024-09-18T01:30:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1561732096 bytes 2024-09-18T01:30:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1603316736 bytes 2024-09-18T01:30:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1667746816 bytes 2024-09-18T01:30:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1706306560 bytes 2024-09-18T01:30:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1724659712 bytes 2024-09-18T01:30:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1764080640 bytes 2024-09-18T01:30:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1789897728 bytes 2024-09-18T01:30:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1824785408 bytes 2024-09-18T01:30:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1900825600 bytes 2024-09-18T01:30:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1961769984 bytes 2024-09-18T01:30:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1996161024 bytes 2024-09-18T01:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2052200448 bytes 2024-09-18T01:30:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2131975168 bytes 2024-09-18T01:30:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2173685760 bytes 2024-09-18T01:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2180283392 bytes 2024-09-18T01:30:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2193062912 bytes 2024-09-18T01:30:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2217038848 bytes 2024-09-18T01:30:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2232653824 bytes 2024-09-18T01:30:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2269688832 bytes 2024-09-18T01:30:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2305897472 bytes 2024-09-18T01:30:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2317100032 bytes 2024-09-18T01:30:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2358010880 bytes 2024-09-18T01:30:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2440144896 bytes 2024-09-18T01:30:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2485261312 bytes 2024-09-18T01:30:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2508942336 bytes 2024-09-18T01:30:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2612247552 bytes 2024-09-18T01:30:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:30:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:30:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:30:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:30:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_18 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-18T01:30:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:30:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2024-09-18T01:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-18T01:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:30:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:30:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-18T01:30:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:30:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_18': 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13474f503af9 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13474f503af9 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:30:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:30:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:55 2024 2024-09-18T01:30:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:55 2024 2024-09-18T01:30:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:55 2024 2024-09-18T01:30:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:30:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:30:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: e28fa577-03c0-4443-8c02-6edcbe8d5994 2024-09-18T01:30:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:30:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:30:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-18T01:30:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:30:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:30:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:30:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:30:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:30:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:30:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:30:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 73808896 bytes 2024-09-18T01:30:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 95115264 bytes 2024-09-18T01:30:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 166879232 bytes 2024-09-18T01:30:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 199844864 bytes 2024-09-18T01:30:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 225654784 bytes 2024-09-18T01:30:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 290991104 bytes 2024-09-18T01:30:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 333083648 bytes 2024-09-18T01:30:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 354988032 bytes 2024-09-18T01:30:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 390025216 bytes 2024-09-18T01:30:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 413724672 bytes 2024-09-18T01:30:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 428564480 bytes 2024-09-18T01:30:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 500783104 bytes 2024-09-18T01:30:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 524498944 bytes 2024-09-18T01:30:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 526494720 bytes 2024-09-18T01:30:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 572561408 bytes 2024-09-18T01:30:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 643833856 bytes 2024-09-18T01:30:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 660956160 bytes 2024-09-18T01:30:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 686068736 bytes 2024-09-18T01:30:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 694196224 bytes 2024-09-18T01:30:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 812749824 bytes 2024-09-18T01:30:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 858527744 bytes 2024-09-18T01:30:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 875801600 bytes 2024-09-18T01:30:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 903181312 bytes 2024-09-18T01:30:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 939171840 bytes 2024-09-18T01:30:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 972175360 bytes 2024-09-18T01:30:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1011685376 bytes 2024-09-18T01:31:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1061776384 bytes 2024-09-18T01:31:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1091391488 bytes 2024-09-18T01:31:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1122811904 bytes 2024-09-18T01:31:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1178707968 bytes 2024-09-18T01:31:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1202049024 bytes 2024-09-18T01:31:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1229464576 bytes 2024-09-18T01:31:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1245815808 bytes 2024-09-18T01:31:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1248271360 bytes 2024-09-18T01:31:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1306918912 bytes 2024-09-18T01:31:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1376759808 bytes 2024-09-18T01:31:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1439758336 bytes 2024-09-18T01:31:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1468047360 bytes 2024-09-18T01:31:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1473651712 bytes 2024-09-18T01:31:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1514363904 bytes 2024-09-18T01:31:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1552038912 bytes 2024-09-18T01:31:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1581050880 bytes 2024-09-18T01:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1600742400 bytes 2024-09-18T01:31:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1633681408 bytes 2024-09-18T01:31:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_19 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:31:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:31:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2024-09-18T01:31:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-18T01:31:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:31:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_19': 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 1353389b4e4c 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.1353389b4e4c 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:01 2024 2024-09-18T01:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:01 2024 2024-09-18T01:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:01 2024 2024-09-18T01:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: fc033df2-2782-4b60-b444-9993613dbf5d 2024-09-18T01:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-18T01:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=14 2024-09-18T01:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:31:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 51997696 bytes 2024-09-18T01:31:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 126273536 bytes 2024-09-18T01:31:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 192229376 bytes 2024-09-18T01:31:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 214764544 bytes 2024-09-18T01:31:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 276020224 bytes 2024-09-18T01:31:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 337304576 bytes 2024-09-18T01:31:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 371862528 bytes 2024-09-18T01:31:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 442269696 bytes 2024-09-18T01:31:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 476350464 bytes 2024-09-18T01:31:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 519656448 bytes 2024-09-18T01:31:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 569129984 bytes 2024-09-18T01:31:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 593956864 bytes 2024-09-18T01:31:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 680588288 bytes 2024-09-18T01:31:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 706146304 bytes 2024-09-18T01:31:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 730186752 bytes 2024-09-18T01:31:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 797139968 bytes 2024-09-18T01:31:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 848125952 bytes 2024-09-18T01:31:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 861934592 bytes 2024-09-18T01:31:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 874221568 bytes 2024-09-18T01:31:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 927382528 bytes 2024-09-18T01:31:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 973380608 bytes 2024-09-18T01:31:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1058500608 bytes 2024-09-18T01:31:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1085795328 bytes 2024-09-18T01:31:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1098773504 bytes 2024-09-18T01:31:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1150417920 bytes 2024-09-18T01:31:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1202058240 bytes 2024-09-18T01:31:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1234030592 bytes 2024-09-18T01:31:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1242398720 bytes 2024-09-18T01:31:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1289414656 bytes 2024-09-18T01:31:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1352561664 bytes 2024-09-18T01:31:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1432876032 bytes 2024-09-18T01:31:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1496630272 bytes 2024-09-18T01:31:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1528849408 bytes 2024-09-18T01:31:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1574406144 bytes 2024-09-18T01:31:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1657091072 bytes 2024-09-18T01:31:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1678658560 bytes 2024-09-18T01:31:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1699948544 bytes 2024-09-18T01:31:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1720931328 bytes 2024-09-18T01:31:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1781554176 bytes 2024-09-18T01:31:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1827371008 bytes 2024-09-18T01:31:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1875406848 bytes 2024-09-18T01:31:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1911227392 bytes 2024-09-18T01:31:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1943834624 bytes 2024-09-18T01:31:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1981519872 bytes 2024-09-18T01:31:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2008640512 bytes 2024-09-18T01:31:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2082176000 bytes 2024-09-18T01:31:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2095441920 bytes 2024-09-18T01:31:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2139143168 bytes 2024-09-18T01:31:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2170114048 bytes 2024-09-18T01:31:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2170941440 bytes 2024-09-18T01:31:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2198508544 bytes 2024-09-18T01:31:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2220813312 bytes 2024-09-18T01:31:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2253209600 bytes 2024-09-18T01:31:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2326512640 bytes 2024-09-18T01:31:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2337186816 bytes 2024-09-18T01:31:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2365165568 bytes 2024-09-18T01:31:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2383432704 bytes 2024-09-18T01:31:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2445317120 bytes 2024-09-18T01:31:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2494328832 bytes 2024-09-18T01:31:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2566337536 bytes 2024-09-18T01:31:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2627538944 bytes 2024-09-18T01:31:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2682661888 bytes 2024-09-18T01:31:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2713144320 bytes 2024-09-18T01:31:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2744979456 bytes 2024-09-18T01:31:10.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2814129152 bytes 2024-09-18T01:31:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2854389760 bytes 2024-09-18T01:31:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2887200768 bytes 2024-09-18T01:31:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2955609088 bytes 2024-09-18T01:31:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2988662784 bytes 2024-09-18T01:31:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3056967680 bytes 2024-09-18T01:31:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3083748352 bytes 2024-09-18T01:31:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3125428224 bytes 2024-09-18T01:31:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3214128128 bytes 2024-09-18T01:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3279671296 bytes 2024-09-18T01:31:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3324549120 bytes 2024-09-18T01:31:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3367426048 bytes 2024-09-18T01:31:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3375153152 bytes 2024-09-18T01:31:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3409822720 bytes 2024-09-18T01:31:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3424561152 bytes 2024-09-18T01:31:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3457662976 bytes 2024-09-18T01:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3493500928 bytes 2024-09-18T01:31:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3539866624 bytes 2024-09-18T01:31:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3565554688 bytes 2024-09-18T01:31:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3661070336 bytes 2024-09-18T01:31:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3697773568 bytes 2024-09-18T01:31:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3706910720 bytes 2024-09-18T01:31:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3731804160 bytes 2024-09-18T01:31:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3743246336 bytes 2024-09-18T01:31:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3766041600 bytes 2024-09-18T01:31:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3800134656 bytes 2024-09-18T01:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3863848960 bytes 2024-09-18T01:31:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3942252544 bytes 2024-09-18T01:31:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4001453056 bytes 2024-09-18T01:31:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4020202496 bytes 2024-09-18T01:31:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4049917952 bytes 2024-09-18T01:31:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4073109504 bytes 2024-09-18T01:31:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4090503168 bytes 2024-09-18T01:31:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4145495040 bytes 2024-09-18T01:31:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4162126848 bytes 2024-09-18T01:31:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4171924480 bytes 2024-09-18T01:31:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4200588288 bytes 2024-09-18T01:31:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4210532352 bytes 2024-09-18T01:31:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4268783616 bytes 2024-09-18T01:31:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 4354065408 bytes 2024-09-18T01:31:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4380297216 bytes 2024-09-18T01:31:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 4418345984 bytes 2024-09-18T01:31:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 4418938880 bytes 2024-09-18T01:31:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 4454084608 bytes 2024-09-18T01:31:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 4463476736 bytes 2024-09-18T01:31:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11000 IOs, 4490603520 bytes 2024-09-18T01:31:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11100 IOs, 4598534144 bytes 2024-09-18T01:31:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11200 IOs, 4648081408 bytes 2024-09-18T01:31:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11300 IOs, 4763924480 bytes 2024-09-18T01:31:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11400 IOs, 4770339840 bytes 2024-09-18T01:31:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11500 IOs, 4783126528 bytes 2024-09-18T01:31:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11600 IOs, 4849688576 bytes 2024-09-18T01:31:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11700 IOs, 4869550080 bytes 2024-09-18T01:31:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:31:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:31:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:31:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:31:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_20 2024-09-18T01:31:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-18T01:31:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:31:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:31:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:31:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:31:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2024-09-18T01:31:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-18T01:31:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-18T01:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_20': 2024-09-18T01:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 135ffb34aa3f 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.135ffb34aa3f 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:31:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:17 2024 2024-09-18T01:31:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:17 2024 2024-09-18T01:31:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:17 2024 2024-09-18T01:31:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:31:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:31:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: b7c93f86-7ae8-4602-a345-dc61cce15b17 2024-09-18T01:31:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:31:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:31:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-18T01:31:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:31:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:31:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=7 2024-09-18T01:31:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:31:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:31:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 27831296 bytes 2024-09-18T01:31:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 65825792 bytes 2024-09-18T01:31:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 77697024 bytes 2024-09-18T01:31:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 95842304 bytes 2024-09-18T01:31:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 144655360 bytes 2024-09-18T01:31:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 211663872 bytes 2024-09-18T01:31:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 238211072 bytes 2024-09-18T01:31:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 282289152 bytes 2024-09-18T01:31:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 318830592 bytes 2024-09-18T01:31:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 355098624 bytes 2024-09-18T01:31:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 392129536 bytes 2024-09-18T01:31:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 400331776 bytes 2024-09-18T01:31:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 410208256 bytes 2024-09-18T01:31:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 446932992 bytes 2024-09-18T01:31:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 450789376 bytes 2024-09-18T01:31:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 491265024 bytes 2024-09-18T01:31:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 540085248 bytes 2024-09-18T01:31:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 551180288 bytes 2024-09-18T01:31:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 574669824 bytes 2024-09-18T01:31:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 628003840 bytes 2024-09-18T01:31:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 704833536 bytes 2024-09-18T01:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 732090368 bytes 2024-09-18T01:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 779390976 bytes 2024-09-18T01:31:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 836339712 bytes 2024-09-18T01:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 877913088 bytes 2024-09-18T01:31:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 912642048 bytes 2024-09-18T01:31:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 941249536 bytes 2024-09-18T01:31:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 959093760 bytes 2024-09-18T01:31:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1000808448 bytes 2024-09-18T01:31:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1037484032 bytes 2024-09-18T01:31:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1118041088 bytes 2024-09-18T01:31:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1168388096 bytes 2024-09-18T01:31:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1208541184 bytes 2024-09-18T01:31:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1229838336 bytes 2024-09-18T01:31:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1247569920 bytes 2024-09-18T01:31:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1265679360 bytes 2024-09-18T01:31:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1312073728 bytes 2024-09-18T01:31:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1347155968 bytes 2024-09-18T01:31:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1368188928 bytes 2024-09-18T01:31:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1413554176 bytes 2024-09-18T01:31:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1491547136 bytes 2024-09-18T01:31:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1500771328 bytes 2024-09-18T01:31:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1531682816 bytes 2024-09-18T01:31:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1587771392 bytes 2024-09-18T01:31:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1627691008 bytes 2024-09-18T01:31:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1709179904 bytes 2024-09-18T01:31:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1753531392 bytes 2024-09-18T01:31:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1789410304 bytes 2024-09-18T01:31:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1805728768 bytes 2024-09-18T01:31:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1828948992 bytes 2024-09-18T01:31:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1847000064 bytes 2024-09-18T01:31:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1870604288 bytes 2024-09-18T01:31:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 1929055232 bytes 2024-09-18T01:31:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 1953126400 bytes 2024-09-18T01:31:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 1988303872 bytes 2024-09-18T01:31:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2008683520 bytes 2024-09-18T01:31:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2090619904 bytes 2024-09-18T01:31:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:31:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:31:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:31:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:31:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_21 2024-09-18T01:31:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:31:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:31:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:31:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:31:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-18T01:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:31:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-18T01:31:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_21': 2024-09-18T01:31:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:31:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:31:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:31:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 136b3385a944 2024-09-18T01:31:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.136b3385a944 2024-09-18T01:31:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:25 2024 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:25 2024 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:25 2024 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 854098b9-42e8-43d8-b6c2-a45666ee66c5 2024-09-18T01:31:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:31:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:31:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-18T01:31:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:31:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:31:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=14 2024-09-18T01:31:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:31:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:31:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 45956096 bytes 2024-09-18T01:31:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 83593216 bytes 2024-09-18T01:31:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 132799488 bytes 2024-09-18T01:31:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 146072576 bytes 2024-09-18T01:31:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 236186624 bytes 2024-09-18T01:31:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 289244160 bytes 2024-09-18T01:31:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 318921728 bytes 2024-09-18T01:31:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 365114368 bytes 2024-09-18T01:31:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 428622848 bytes 2024-09-18T01:31:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 483655680 bytes 2024-09-18T01:31:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 525872128 bytes 2024-09-18T01:31:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 558744576 bytes 2024-09-18T01:31:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 623496192 bytes 2024-09-18T01:31:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 672094208 bytes 2024-09-18T01:31:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 734150656 bytes 2024-09-18T01:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 800515072 bytes 2024-09-18T01:31:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 886196224 bytes 2024-09-18T01:31:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 937138176 bytes 2024-09-18T01:31:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 950973440 bytes 2024-09-18T01:31:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 992653312 bytes 2024-09-18T01:31:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1015117824 bytes 2024-09-18T01:31:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1096542208 bytes 2024-09-18T01:31:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1132350464 bytes 2024-09-18T01:31:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1216099328 bytes 2024-09-18T01:31:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1275835392 bytes 2024-09-18T01:31:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1360145408 bytes 2024-09-18T01:31:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1392118784 bytes 2024-09-18T01:31:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1397417984 bytes 2024-09-18T01:31:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1429694464 bytes 2024-09-18T01:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1472496640 bytes 2024-09-18T01:31:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1473120256 bytes 2024-09-18T01:31:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1521459200 bytes 2024-09-18T01:31:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1605855232 bytes 2024-09-18T01:31:30.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1657918464 bytes 2024-09-18T01:31:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1658534912 bytes 2024-09-18T01:31:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1724612608 bytes 2024-09-18T01:31:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1770508288 bytes 2024-09-18T01:31:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1793012736 bytes 2024-09-18T01:31:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1827298304 bytes 2024-09-18T01:31:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1888118784 bytes 2024-09-18T01:31:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1927109632 bytes 2024-09-18T01:31:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1944150016 bytes 2024-09-18T01:31:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1988495360 bytes 2024-09-18T01:31:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2042730496 bytes 2024-09-18T01:31:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2079425536 bytes 2024-09-18T01:31:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2125785088 bytes 2024-09-18T01:31:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2151548928 bytes 2024-09-18T01:31:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2165382144 bytes 2024-09-18T01:31:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2180286464 bytes 2024-09-18T01:31:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2211585024 bytes 2024-09-18T01:31:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2245897216 bytes 2024-09-18T01:31:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2246529024 bytes 2024-09-18T01:31:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2342597632 bytes 2024-09-18T01:31:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2361469952 bytes 2024-09-18T01:31:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2388499456 bytes 2024-09-18T01:31:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2497268736 bytes 2024-09-18T01:31:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2609185792 bytes 2024-09-18T01:31:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2646661120 bytes 2024-09-18T01:31:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2737500160 bytes 2024-09-18T01:31:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2741404672 bytes 2024-09-18T01:31:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2791841792 bytes 2024-09-18T01:31:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2829489152 bytes 2024-09-18T01:31:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2903378944 bytes 2024-09-18T01:31:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2965126144 bytes 2024-09-18T01:31:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 3002098688 bytes 2024-09-18T01:31:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 3044599808 bytes 2024-09-18T01:31:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3071683584 bytes 2024-09-18T01:31:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3150078976 bytes 2024-09-18T01:31:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3166313472 bytes 2024-09-18T01:31:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3192050688 bytes 2024-09-18T01:31:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3224467456 bytes 2024-09-18T01:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3290005504 bytes 2024-09-18T01:31:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3353954304 bytes 2024-09-18T01:31:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3373706240 bytes 2024-09-18T01:31:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3417479168 bytes 2024-09-18T01:31:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3506819072 bytes 2024-09-18T01:31:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3552879616 bytes 2024-09-18T01:31:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3632267264 bytes 2024-09-18T01:31:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3683198976 bytes 2024-09-18T01:31:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3766344704 bytes 2024-09-18T01:31:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3785458688 bytes 2024-09-18T01:31:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3875911680 bytes 2024-09-18T01:31:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3923680256 bytes 2024-09-18T01:31:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3990209536 bytes 2024-09-18T01:31:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 4051183616 bytes 2024-09-18T01:31:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 4063680512 bytes 2024-09-18T01:31:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 4077418496 bytes 2024-09-18T01:31:36.936 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4437880405967859) > weight (0.25) 2024-09-18T01:31:36.936 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.40407659630006243) > weight (0.25) 2024-09-18T01:31:36.936 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6633280007906492) > weight (0.25) 2024-09-18T01:31:36.936 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6259568413951442) > weight (0.25) 2024-09-18T01:31:36.936 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-18T01:31:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 4124258304 bytes 2024-09-18T01:31:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 4133037056 bytes 2024-09-18T01:31:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 4162672640 bytes 2024-09-18T01:31:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4181114880 bytes 2024-09-18T01:31:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4217276416 bytes 2024-09-18T01:31:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4276413440 bytes 2024-09-18T01:31:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4351006720 bytes 2024-09-18T01:31:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4365579264 bytes 2024-09-18T01:31:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4424010752 bytes 2024-09-18T01:31:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4527192064 bytes 2024-09-18T01:31:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4532012032 bytes 2024-09-18T01:31:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4558533632 bytes 2024-09-18T01:31:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4606806016 bytes 2024-09-18T01:31:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4621711360 bytes 2024-09-18T01:31:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4702469120 bytes 2024-09-18T01:31:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4743608320 bytes 2024-09-18T01:31:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 4783822848 bytes 2024-09-18T01:31:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4828445696 bytes 2024-09-18T01:31:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 4873020416 bytes 2024-09-18T01:31:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 4963833856 bytes 2024-09-18T01:31:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 4967952384 bytes 2024-09-18T01:31:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 5025931264 bytes 2024-09-18T01:31:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11000 IOs, 5109337088 bytes 2024-09-18T01:31:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11100 IOs, 5203979264 bytes 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_22 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:31:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2024-09-18T01:31:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:31:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:31:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-18T01:31:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:31:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-18T01:31:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:31:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-18T01:31:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_22': 2024-09-18T01:31:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:31:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:31:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:31:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 137746e88aa0 2024-09-18T01:31:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.137746e88aa0 2024-09-18T01:31:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:40 2024 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:40 2024 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:40 2024 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: b41be30f-1770-49fc-9175-aad90b2449de 2024-09-18T01:31:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:31:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:31:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-18T01:31:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:31:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:31:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:31:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:31:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:31:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 51106816 bytes 2024-09-18T01:31:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 67529728 bytes 2024-09-18T01:31:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 112208896 bytes 2024-09-18T01:31:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 202040320 bytes 2024-09-18T01:31:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 236578816 bytes 2024-09-18T01:31:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 237205504 bytes 2024-09-18T01:31:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 294447104 bytes 2024-09-18T01:31:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 330783744 bytes 2024-09-18T01:31:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 398982144 bytes 2024-09-18T01:31:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 460795904 bytes 2024-09-18T01:31:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 483275776 bytes 2024-09-18T01:31:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 536226816 bytes 2024-09-18T01:31:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 539940864 bytes 2024-09-18T01:31:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 560694272 bytes 2024-09-18T01:31:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 635194368 bytes 2024-09-18T01:31:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 658546688 bytes 2024-09-18T01:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 679805952 bytes 2024-09-18T01:31:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 718362624 bytes 2024-09-18T01:31:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 822867968 bytes 2024-09-18T01:31:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 897974272 bytes 2024-09-18T01:31:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 901097472 bytes 2024-09-18T01:31:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 946894848 bytes 2024-09-18T01:31:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1004140544 bytes 2024-09-18T01:31:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1086519296 bytes 2024-09-18T01:31:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1133753344 bytes 2024-09-18T01:31:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1237875712 bytes 2024-09-18T01:31:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1258535936 bytes 2024-09-18T01:31:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1296270336 bytes 2024-09-18T01:31:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1390287872 bytes 2024-09-18T01:31:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1433395200 bytes 2024-09-18T01:31:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1498037248 bytes 2024-09-18T01:31:45.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1586693120 bytes 2024-09-18T01:31:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1611635712 bytes 2024-09-18T01:31:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1653020672 bytes 2024-09-18T01:31:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1695729664 bytes 2024-09-18T01:31:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1758129152 bytes 2024-09-18T01:31:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1792173056 bytes 2024-09-18T01:31:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1840277504 bytes 2024-09-18T01:31:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1970136064 bytes 2024-09-18T01:31:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1986661376 bytes 2024-09-18T01:31:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 2101045248 bytes 2024-09-18T01:31:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2101650432 bytes 2024-09-18T01:31:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2124461056 bytes 2024-09-18T01:31:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2125080576 bytes 2024-09-18T01:31:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2170104832 bytes 2024-09-18T01:31:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2239070208 bytes 2024-09-18T01:31:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_23 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-18T01:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:31:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-18T01:31:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:31:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-18T01:31:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_23': 2024-09-18T01:31:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:31:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 138351fa4a0d 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.138351fa4a0d 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:47 2024 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:47 2024 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:47 2024 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:31:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 0b0b9342-5291-4bb5-ade8-4fa8d9ff735c 2024-09-18T01:31:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:31:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:31:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-18T01:31:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:31:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:31:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=8 2024-09-18T01:31:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:31:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:31:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 60625920 bytes 2024-09-18T01:31:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 86097920 bytes 2024-09-18T01:31:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 86709248 bytes 2024-09-18T01:31:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 154950656 bytes 2024-09-18T01:31:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 168671232 bytes 2024-09-18T01:31:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 189726720 bytes 2024-09-18T01:31:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 199348224 bytes 2024-09-18T01:31:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 267017216 bytes 2024-09-18T01:31:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 320418816 bytes 2024-09-18T01:31:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 371708928 bytes 2024-09-18T01:31:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 385623040 bytes 2024-09-18T01:31:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 444835840 bytes 2024-09-18T01:31:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 445462528 bytes 2024-09-18T01:31:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 493976576 bytes 2024-09-18T01:31:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 548102144 bytes 2024-09-18T01:31:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 598457344 bytes 2024-09-18T01:31:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 622170112 bytes 2024-09-18T01:31:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 669155328 bytes 2024-09-18T01:31:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 705029120 bytes 2024-09-18T01:31:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 752510976 bytes 2024-09-18T01:31:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 797079552 bytes 2024-09-18T01:31:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 894374912 bytes 2024-09-18T01:31:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 935321600 bytes 2024-09-18T01:31:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 996381696 bytes 2024-09-18T01:31:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1023442944 bytes 2024-09-18T01:31:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1082588160 bytes 2024-09-18T01:31:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1132676096 bytes 2024-09-18T01:31:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1194752000 bytes 2024-09-18T01:31:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1269456896 bytes 2024-09-18T01:31:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1306663936 bytes 2024-09-18T01:31:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1340724224 bytes 2024-09-18T01:31:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1359305728 bytes 2024-09-18T01:31:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1457058816 bytes 2024-09-18T01:31:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1460701184 bytes 2024-09-18T01:31:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1514391552 bytes 2024-09-18T01:31:52.721 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5686261929757892) > weight (0.25) 2024-09-18T01:31:52.722 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-18T01:31:52.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-18T01:31:52.722 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-18T01:31:52.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:31:52.722+0000 7fc33d6d2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 41527) UID: 1000 2024-09-18T01:31:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1527178240 bytes 2024-09-18T01:31:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1567841280 bytes 2024-09-18T01:31:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1596541952 bytes 2024-09-18T01:31:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1661177856 bytes 2024-09-18T01:31:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1676587008 bytes 2024-09-18T01:31:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1677196288 bytes 2024-09-18T01:31:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1705241600 bytes 2024-09-18T01:31:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1735532544 bytes 2024-09-18T01:31:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1811310592 bytes 2024-09-18T01:31:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1846971392 bytes 2024-09-18T01:31:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1893339136 bytes 2024-09-18T01:31:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1918865408 bytes 2024-09-18T01:31:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1939060736 bytes 2024-09-18T01:31:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1971778560 bytes 2024-09-18T01:31:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2046875648 bytes 2024-09-18T01:31:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2158615552 bytes 2024-09-18T01:31:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2217329664 bytes 2024-09-18T01:31:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2278631424 bytes 2024-09-18T01:31:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2295999488 bytes 2024-09-18T01:31:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2349727744 bytes 2024-09-18T01:31:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2387637248 bytes 2024-09-18T01:31:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2488031232 bytes 2024-09-18T01:31:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2543415296 bytes 2024-09-18T01:31:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2634157056 bytes 2024-09-18T01:31:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2669587456 bytes 2024-09-18T01:31:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2709278720 bytes 2024-09-18T01:31:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_24 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:31:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:31:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:31:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2024-09-18T01:31:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:31:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:31:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:31:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-18T01:31:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:31:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:31:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-18T01:31:56.723 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-18T01:31:56.723 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:31:56.723 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-18T01:31:56.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-18T01:31:56.724 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-18T01:31:56.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-18T01:31:56.727 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-18T01:31:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:31:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-18T01:31:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_24': 2024-09-18T01:31:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 138f75e6cef4 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.138f75e6cef4 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:56 2024 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:56 2024 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:56 2024 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:31:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:31:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 6425ebaf-56a7-47a1-9547-1c0136cbba11 2024-09-18T01:31:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:31:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:31:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-18T01:31:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:31:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:31:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:31:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:31:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=14 2024-09-18T01:31:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:31:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:31:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 51015680 bytes 2024-09-18T01:31:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 78297088 bytes 2024-09-18T01:31:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 162125824 bytes 2024-09-18T01:31:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 171898880 bytes 2024-09-18T01:31:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 197656576 bytes 2024-09-18T01:31:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 229948416 bytes 2024-09-18T01:31:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 255496192 bytes 2024-09-18T01:31:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 296781824 bytes 2024-09-18T01:31:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 344772608 bytes 2024-09-18T01:31:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 395395072 bytes 2024-09-18T01:31:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 441915392 bytes 2024-09-18T01:31:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 462230528 bytes 2024-09-18T01:31:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 569611264 bytes 2024-09-18T01:31:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 598441984 bytes 2024-09-18T01:31:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 628643840 bytes 2024-09-18T01:31:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 644123648 bytes 2024-09-18T01:31:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 663782400 bytes 2024-09-18T01:31:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 700163072 bytes 2024-09-18T01:31:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 720832512 bytes 2024-09-18T01:31:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 820180992 bytes 2024-09-18T01:31:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 878159872 bytes 2024-09-18T01:31:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 953863168 bytes 2024-09-18T01:31:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 998563840 bytes 2024-09-18T01:32:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1048080384 bytes 2024-09-18T01:32:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1120121856 bytes 2024-09-18T01:32:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1162321920 bytes 2024-09-18T01:32:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1285908480 bytes 2024-09-18T01:32:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1295479808 bytes 2024-09-18T01:32:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1383514112 bytes 2024-09-18T01:32:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1384108032 bytes 2024-09-18T01:32:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1421175808 bytes 2024-09-18T01:32:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1480849408 bytes 2024-09-18T01:32:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1511126016 bytes 2024-09-18T01:32:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1559473152 bytes 2024-09-18T01:32:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1583289344 bytes 2024-09-18T01:32:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1587369984 bytes 2024-09-18T01:32:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1652082688 bytes 2024-09-18T01:32:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1656596480 bytes 2024-09-18T01:32:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1724934144 bytes 2024-09-18T01:32:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1796256768 bytes 2024-09-18T01:32:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1856952320 bytes 2024-09-18T01:32:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1886453760 bytes 2024-09-18T01:32:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1935941632 bytes 2024-09-18T01:32:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1959467008 bytes 2024-09-18T01:32:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1992540160 bytes 2024-09-18T01:32:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2001470464 bytes 2024-09-18T01:32:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2073753600 bytes 2024-09-18T01:32:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2169160704 bytes 2024-09-18T01:32:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2189485056 bytes 2024-09-18T01:32:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2230282240 bytes 2024-09-18T01:32:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2282978304 bytes 2024-09-18T01:32:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2309744640 bytes 2024-09-18T01:32:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2310378496 bytes 2024-09-18T01:32:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2311440384 bytes 2024-09-18T01:32:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2417287168 bytes 2024-09-18T01:32:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2417895424 bytes 2024-09-18T01:32:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2471560192 bytes 2024-09-18T01:32:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2472190976 bytes 2024-09-18T01:32:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2552078336 bytes 2024-09-18T01:32:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2583676928 bytes 2024-09-18T01:32:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2638527488 bytes 2024-09-18T01:32:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2642768896 bytes 2024-09-18T01:32:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2719210496 bytes 2024-09-18T01:32:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2758843392 bytes 2024-09-18T01:32:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2816373760 bytes 2024-09-18T01:32:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2852993024 bytes 2024-09-18T01:32:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2892128256 bytes 2024-09-18T01:32:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2919142400 bytes 2024-09-18T01:32:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2943740928 bytes 2024-09-18T01:32:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2959646720 bytes 2024-09-18T01:32:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3013743616 bytes 2024-09-18T01:32:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3071397888 bytes 2024-09-18T01:32:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3104610304 bytes 2024-09-18T01:32:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3173164032 bytes 2024-09-18T01:32:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3206071296 bytes 2024-09-18T01:32:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3260858368 bytes 2024-09-18T01:32:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3261474816 bytes 2024-09-18T01:32:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3336892416 bytes 2024-09-18T01:32:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3383553024 bytes 2024-09-18T01:32:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3437284352 bytes 2024-09-18T01:32:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3492217856 bytes 2024-09-18T01:32:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3534021632 bytes 2024-09-18T01:32:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3544811520 bytes 2024-09-18T01:32:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3659202560 bytes 2024-09-18T01:32:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3703172096 bytes 2024-09-18T01:32:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3721712640 bytes 2024-09-18T01:32:07.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3743906816 bytes 2024-09-18T01:32:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3820123136 bytes 2024-09-18T01:32:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3910041600 bytes 2024-09-18T01:32:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3962622976 bytes 2024-09-18T01:32:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3999582208 bytes 2024-09-18T01:32:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4031006720 bytes 2024-09-18T01:32:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4118735872 bytes 2024-09-18T01:32:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4141941760 bytes 2024-09-18T01:32:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4148126720 bytes 2024-09-18T01:32:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4238356480 bytes 2024-09-18T01:32:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4243454976 bytes 2024-09-18T01:32:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4260964352 bytes 2024-09-18T01:32:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4292412416 bytes 2024-09-18T01:32:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4333786112 bytes 2024-09-18T01:32:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4421015552 bytes 2024-09-18T01:32:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4477768704 bytes 2024-09-18T01:32:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4572286976 bytes 2024-09-18T01:32:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 4619286528 bytes 2024-09-18T01:32:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4644566016 bytes 2024-09-18T01:32:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 4670015488 bytes 2024-09-18T01:32:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 4785930240 bytes 2024-09-18T01:32:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 4824314880 bytes 2024-09-18T01:32:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 4904220672 bytes 2024-09-18T01:32:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11000 IOs, 4999026688 bytes 2024-09-18T01:32:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11100 IOs, 5030528000 bytes 2024-09-18T01:32:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:32:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:32:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:32:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_25 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:32:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:32:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2024-09-18T01:32:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-18T01:32:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:32:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-18T01:32:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_25': 2024-09-18T01:32:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:32:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:32:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13a1e93415b5 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13a1e93415b5 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:11 2024 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:11 2024 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:11 2024 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 0ee8e883-4767-4c92-9030-1bb0af68068c 2024-09-18T01:32:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:32:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:32:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-18T01:32:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:32:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:32:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:32:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:32:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:32:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 73537536 bytes 2024-09-18T01:32:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 74150912 bytes 2024-09-18T01:32:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 109548544 bytes 2024-09-18T01:32:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 192197632 bytes 2024-09-18T01:32:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 266707968 bytes 2024-09-18T01:32:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 316971008 bytes 2024-09-18T01:32:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 402444288 bytes 2024-09-18T01:32:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 421687296 bytes 2024-09-18T01:32:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 464674816 bytes 2024-09-18T01:32:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 502686720 bytes 2024-09-18T01:32:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 533967872 bytes 2024-09-18T01:32:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 534586368 bytes 2024-09-18T01:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 609808384 bytes 2024-09-18T01:32:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 673146880 bytes 2024-09-18T01:32:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 735388672 bytes 2024-09-18T01:32:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 823318528 bytes 2024-09-18T01:32:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 837774336 bytes 2024-09-18T01:32:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 941598720 bytes 2024-09-18T01:32:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 1000933376 bytes 2024-09-18T01:32:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1036830720 bytes 2024-09-18T01:32:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1092859904 bytes 2024-09-18T01:32:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1135473664 bytes 2024-09-18T01:32:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1186173952 bytes 2024-09-18T01:32:15.238 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:32:15.237+0000 7f78fa24b640 -1 reset not still connected to 0x55b9ddd04680 2024-09-18T01:32:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1219177472 bytes 2024-09-18T01:32:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1235765248 bytes 2024-09-18T01:32:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1297471488 bytes 2024-09-18T01:32:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1366792192 bytes 2024-09-18T01:32:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1410050048 bytes 2024-09-18T01:32:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1476847616 bytes 2024-09-18T01:32:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1496966144 bytes 2024-09-18T01:32:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1515791360 bytes 2024-09-18T01:32:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1550743552 bytes 2024-09-18T01:32:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1555141632 bytes 2024-09-18T01:32:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1617336320 bytes 2024-09-18T01:32:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1656893440 bytes 2024-09-18T01:32:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1668040704 bytes 2024-09-18T01:32:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1781045248 bytes 2024-09-18T01:32:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1797566464 bytes 2024-09-18T01:32:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1877221376 bytes 2024-09-18T01:32:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1981293568 bytes 2024-09-18T01:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 2022797312 bytes 2024-09-18T01:32:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2074445824 bytes 2024-09-18T01:32:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2127684608 bytes 2024-09-18T01:32:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2169663488 bytes 2024-09-18T01:32:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2185129984 bytes 2024-09-18T01:32:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2200812544 bytes 2024-09-18T01:32:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:32:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:32:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:32:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:32:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_26 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-18T01:32:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2024-09-18T01:32:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:32:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:32:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-18T01:32:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:32:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-18T01:32:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:32:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-18T01:32:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_26': 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13adb412bf7b 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13adb412bf7b 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:18 2024 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:18 2024 2024-09-18T01:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:18 2024 2024-09-18T01:32:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:32:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:32:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 41d5e821-71ed-41c3-afd1-9e875de13804 2024-09-18T01:32:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:32:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:32:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-18T01:32:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:32:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:32:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=13 2024-09-18T01:32:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:32:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:32:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 64422912 bytes 2024-09-18T01:32:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 96262144 bytes 2024-09-18T01:32:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 133775360 bytes 2024-09-18T01:32:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 172045312 bytes 2024-09-18T01:32:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 211721216 bytes 2024-09-18T01:32:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 214494208 bytes 2024-09-18T01:32:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 242600960 bytes 2024-09-18T01:32:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 270663680 bytes 2024-09-18T01:32:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 296276992 bytes 2024-09-18T01:32:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 319803392 bytes 2024-09-18T01:32:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 383300608 bytes 2024-09-18T01:32:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 408962048 bytes 2024-09-18T01:32:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 441743360 bytes 2024-09-18T01:32:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 528220160 bytes 2024-09-18T01:32:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 558215168 bytes 2024-09-18T01:32:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 632901632 bytes 2024-09-18T01:32:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 704020480 bytes 2024-09-18T01:32:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 709477376 bytes 2024-09-18T01:32:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 768332800 bytes 2024-09-18T01:32:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 775561216 bytes 2024-09-18T01:32:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 805539840 bytes 2024-09-18T01:32:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 827633664 bytes 2024-09-18T01:32:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 828252160 bytes 2024-09-18T01:32:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 846858240 bytes 2024-09-18T01:32:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 883517440 bytes 2024-09-18T01:32:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 939166720 bytes 2024-09-18T01:32:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 939792384 bytes 2024-09-18T01:32:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 973111296 bytes 2024-09-18T01:32:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1021997056 bytes 2024-09-18T01:32:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1058130944 bytes 2024-09-18T01:32:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1075529728 bytes 2024-09-18T01:32:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1133159424 bytes 2024-09-18T01:32:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1192074240 bytes 2024-09-18T01:32:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1265344512 bytes 2024-09-18T01:32:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1359799296 bytes 2024-09-18T01:32:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1398128640 bytes 2024-09-18T01:32:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1444111360 bytes 2024-09-18T01:32:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1477920768 bytes 2024-09-18T01:32:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1489031168 bytes 2024-09-18T01:32:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1537015808 bytes 2024-09-18T01:32:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1549589504 bytes 2024-09-18T01:32:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1600545792 bytes 2024-09-18T01:32:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1670145024 bytes 2024-09-18T01:32:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1708702720 bytes 2024-09-18T01:32:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1766051840 bytes 2024-09-18T01:32:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1769100288 bytes 2024-09-18T01:32:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1799704576 bytes 2024-09-18T01:32:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1870945280 bytes 2024-09-18T01:32:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1874738176 bytes 2024-09-18T01:32:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1900517376 bytes 2024-09-18T01:32:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1933764608 bytes 2024-09-18T01:32:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1979998208 bytes 2024-09-18T01:32:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2030510080 bytes 2024-09-18T01:32:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2066749440 bytes 2024-09-18T01:32:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2098995200 bytes 2024-09-18T01:32:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2158621696 bytes 2024-09-18T01:32:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2199335936 bytes 2024-09-18T01:32:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2257580032 bytes 2024-09-18T01:32:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2269053952 bytes 2024-09-18T01:32:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2303546368 bytes 2024-09-18T01:32:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2377820160 bytes 2024-09-18T01:32:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2401206272 bytes 2024-09-18T01:32:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2415709184 bytes 2024-09-18T01:32:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2423096320 bytes 2024-09-18T01:32:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2484895744 bytes 2024-09-18T01:32:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2531381248 bytes 2024-09-18T01:32:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2597457920 bytes 2024-09-18T01:32:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2649677824 bytes 2024-09-18T01:32:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2685205504 bytes 2024-09-18T01:32:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2752010240 bytes 2024-09-18T01:32:27.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2799538176 bytes 2024-09-18T01:32:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2829786112 bytes 2024-09-18T01:32:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2875965440 bytes 2024-09-18T01:32:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 2899999744 bytes 2024-09-18T01:32:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 2938164224 bytes 2024-09-18T01:32:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3016437760 bytes 2024-09-18T01:32:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3079075840 bytes 2024-09-18T01:32:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3116399616 bytes 2024-09-18T01:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3159001088 bytes 2024-09-18T01:32:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3237632000 bytes 2024-09-18T01:32:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3284190208 bytes 2024-09-18T01:32:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3325328384 bytes 2024-09-18T01:32:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3368418304 bytes 2024-09-18T01:32:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3412343808 bytes 2024-09-18T01:32:29.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3447998464 bytes 2024-09-18T01:32:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3456091136 bytes 2024-09-18T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3493597184 bytes 2024-09-18T01:32:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3577123840 bytes 2024-09-18T01:32:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3640026112 bytes 2024-09-18T01:32:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3681188864 bytes 2024-09-18T01:32:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3714495488 bytes 2024-09-18T01:32:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3760710656 bytes 2024-09-18T01:32:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 3865762816 bytes 2024-09-18T01:32:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 3927454720 bytes 2024-09-18T01:32:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 3935678464 bytes 2024-09-18T01:32:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4008492032 bytes 2024-09-18T01:32:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4045080576 bytes 2024-09-18T01:32:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4075015168 bytes 2024-09-18T01:32:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4152969216 bytes 2024-09-18T01:32:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4174970880 bytes 2024-09-18T01:32:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4241497088 bytes 2024-09-18T01:32:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4276678656 bytes 2024-09-18T01:32:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4373535744 bytes 2024-09-18T01:32:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 4411537408 bytes 2024-09-18T01:32:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4460727296 bytes 2024-09-18T01:32:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:32:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_27 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:32:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-18T01:32:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:32:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-18T01:32:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_27': 2024-09-18T01:32:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:32:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:32:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:32:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13b92c5b33e9 2024-09-18T01:32:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13b92c5b33e9 2024-09-18T01:32:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:32 2024 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:32 2024 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:32 2024 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: ba4cfda3-b609-4e89-9671-ed59bd54d468 2024-09-18T01:32:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:32:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:32:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-18T01:32:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:32:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:32:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=8 2024-09-18T01:32:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:32:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:32:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 617472 bytes 2024-09-18T01:32:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 1245184 bytes 2024-09-18T01:32:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 27762688 bytes 2024-09-18T01:32:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 42838016 bytes 2024-09-18T01:32:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 64374784 bytes 2024-09-18T01:32:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 138423296 bytes 2024-09-18T01:32:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 154130432 bytes 2024-09-18T01:32:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 158337024 bytes 2024-09-18T01:32:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 240424960 bytes 2024-09-18T01:32:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 241036288 bytes 2024-09-18T01:32:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 241634304 bytes 2024-09-18T01:32:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 282862592 bytes 2024-09-18T01:32:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 355369984 bytes 2024-09-18T01:32:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 372409344 bytes 2024-09-18T01:32:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 382139392 bytes 2024-09-18T01:32:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 396893184 bytes 2024-09-18T01:32:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 430802944 bytes 2024-09-18T01:32:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 493956096 bytes 2024-09-18T01:32:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 532976640 bytes 2024-09-18T01:32:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 533603328 bytes 2024-09-18T01:32:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 583201792 bytes 2024-09-18T01:32:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 691906560 bytes 2024-09-18T01:32:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 753823744 bytes 2024-09-18T01:32:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 760030208 bytes 2024-09-18T01:32:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 802100224 bytes 2024-09-18T01:32:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 824433664 bytes 2024-09-18T01:32:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 870481920 bytes 2024-09-18T01:32:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 894380032 bytes 2024-09-18T01:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 903159808 bytes 2024-09-18T01:32:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 982790144 bytes 2024-09-18T01:32:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1048884224 bytes 2024-09-18T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1077510144 bytes 2024-09-18T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1098329088 bytes 2024-09-18T01:32:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1107040256 bytes 2024-09-18T01:32:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1118284800 bytes 2024-09-18T01:32:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1173352448 bytes 2024-09-18T01:32:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1203423232 bytes 2024-09-18T01:32:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1228375040 bytes 2024-09-18T01:32:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1257416704 bytes 2024-09-18T01:32:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1262923776 bytes 2024-09-18T01:32:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1312987136 bytes 2024-09-18T01:32:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1403022336 bytes 2024-09-18T01:32:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1450027008 bytes 2024-09-18T01:32:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1493618688 bytes 2024-09-18T01:32:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1612148736 bytes 2024-09-18T01:32:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1665718272 bytes 2024-09-18T01:32:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1691340800 bytes 2024-09-18T01:32:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1724945408 bytes 2024-09-18T01:32:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1751567360 bytes 2024-09-18T01:32:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1768824832 bytes 2024-09-18T01:32:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1808418816 bytes 2024-09-18T01:32:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1853186048 bytes 2024-09-18T01:32:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 1855841280 bytes 2024-09-18T01:32:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 1868634112 bytes 2024-09-18T01:32:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 1919600640 bytes 2024-09-18T01:32:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 1990173696 bytes 2024-09-18T01:32:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2109687808 bytes 2024-09-18T01:32:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2128442368 bytes 2024-09-18T01:32:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2155045888 bytes 2024-09-18T01:32:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2212542464 bytes 2024-09-18T01:32:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2252011520 bytes 2024-09-18T01:32:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2347931648 bytes 2024-09-18T01:32:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2389568512 bytes 2024-09-18T01:32:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2442562560 bytes 2024-09-18T01:32:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2549857280 bytes 2024-09-18T01:32:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_28 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2024-09-18T01:32:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:32:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:32:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-18T01:32:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:32:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-18T01:32:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:32:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-18T01:32:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_28': 2024-09-18T01:32:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13c5e856034e 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13c5e856034e 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:41 2024 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:41 2024 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:41 2024 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:32:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 7ba84989-9e9f-431c-9a90-7999790fff88 2024-09-18T01:32:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:32:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:32:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-18T01:32:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:32:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:32:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:32:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:32:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:32:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 34226176 bytes 2024-09-18T01:32:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 99471360 bytes 2024-09-18T01:32:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 124262400 bytes 2024-09-18T01:32:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 153335808 bytes 2024-09-18T01:32:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 162704384 bytes 2024-09-18T01:32:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 266212352 bytes 2024-09-18T01:32:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 313038848 bytes 2024-09-18T01:32:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 332120064 bytes 2024-09-18T01:32:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 366051328 bytes 2024-09-18T01:32:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 405111808 bytes 2024-09-18T01:32:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 456497152 bytes 2024-09-18T01:32:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 508347392 bytes 2024-09-18T01:32:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 541165568 bytes 2024-09-18T01:32:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 572587008 bytes 2024-09-18T01:32:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 601218048 bytes 2024-09-18T01:32:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 654493696 bytes 2024-09-18T01:32:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 655088640 bytes 2024-09-18T01:32:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 682049536 bytes 2024-09-18T01:32:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 729104384 bytes 2024-09-18T01:32:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 770949120 bytes 2024-09-18T01:32:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 797249536 bytes 2024-09-18T01:32:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 821827584 bytes 2024-09-18T01:32:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 849925120 bytes 2024-09-18T01:32:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 850523136 bytes 2024-09-18T01:32:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 932846592 bytes 2024-09-18T01:32:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 933486592 bytes 2024-09-18T01:32:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1011962880 bytes 2024-09-18T01:32:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1019211776 bytes 2024-09-18T01:32:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1095782400 bytes 2024-09-18T01:32:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1140426752 bytes 2024-09-18T01:32:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1218918400 bytes 2024-09-18T01:32:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1253372928 bytes 2024-09-18T01:32:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1253988352 bytes 2024-09-18T01:32:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1289772032 bytes 2024-09-18T01:32:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1354857472 bytes 2024-09-18T01:32:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1359768576 bytes 2024-09-18T01:32:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1416705024 bytes 2024-09-18T01:32:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1483451392 bytes 2024-09-18T01:32:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1520954368 bytes 2024-09-18T01:32:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1622224896 bytes 2024-09-18T01:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_29 2024-09-18T01:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-18T01:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-18T01:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:32:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_29': 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13d16ee0e8d4 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13d16ee0e8d4 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:47 2024 2024-09-18T01:32:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:47 2024 2024-09-18T01:32:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:47 2024 2024-09-18T01:32:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:32:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:32:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 044f2e71-cbaa-44f3-9d02-93b1935ce488 2024-09-18T01:32:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:32:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:32:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-18T01:32:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:32:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:32:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:32:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:32:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:32:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 49484800 bytes 2024-09-18T01:32:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 135348224 bytes 2024-09-18T01:32:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 173976576 bytes 2024-09-18T01:32:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 196325376 bytes 2024-09-18T01:32:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 243464192 bytes 2024-09-18T01:32:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 272100352 bytes 2024-09-18T01:32:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 309551104 bytes 2024-09-18T01:32:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 313967616 bytes 2024-09-18T01:32:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 316761088 bytes 2024-09-18T01:32:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 369053696 bytes 2024-09-18T01:32:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 380286976 bytes 2024-09-18T01:32:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 440359936 bytes 2024-09-18T01:32:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 525454336 bytes 2024-09-18T01:32:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 588801024 bytes 2024-09-18T01:32:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 609229824 bytes 2024-09-18T01:32:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 642725888 bytes 2024-09-18T01:32:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 671536128 bytes 2024-09-18T01:32:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 747310080 bytes 2024-09-18T01:32:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 778758144 bytes 2024-09-18T01:32:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 787887104 bytes 2024-09-18T01:32:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 814385152 bytes 2024-09-18T01:32:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 870328320 bytes 2024-09-18T01:32:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 923878400 bytes 2024-09-18T01:32:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 951872512 bytes 2024-09-18T01:32:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 980222976 bytes 2024-09-18T01:32:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 993821696 bytes 2024-09-18T01:32:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1008567296 bytes 2024-09-18T01:32:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1025123328 bytes 2024-09-18T01:32:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1111113728 bytes 2024-09-18T01:32:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1164574720 bytes 2024-09-18T01:32:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1184776192 bytes 2024-09-18T01:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1235525632 bytes 2024-09-18T01:32:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1328418816 bytes 2024-09-18T01:32:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1388890112 bytes 2024-09-18T01:32:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1407486976 bytes 2024-09-18T01:32:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1469067264 bytes 2024-09-18T01:32:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1507153920 bytes 2024-09-18T01:32:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1544169472 bytes 2024-09-18T01:32:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1632540672 bytes 2024-09-18T01:32:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1633168384 bytes 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_30 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:32:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:32:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:32:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2024-09-18T01:32:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-18T01:32:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:32:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-18T01:32:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_30': 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13dd6799f0dd 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13dd6799f0dd 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:53 2024 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:53 2024 2024-09-18T01:32:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:53 2024 2024-09-18T01:32:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:32:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:32:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 82774ab4-7663-4c2d-b86c-48df151df61b 2024-09-18T01:32:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:32:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:32:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-18T01:32:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:32:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:32:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:32:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:32:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:32:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:32:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 25101312 bytes 2024-09-18T01:32:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 99556352 bytes 2024-09-18T01:32:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 200824832 bytes 2024-09-18T01:32:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 260364288 bytes 2024-09-18T01:32:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 302289920 bytes 2024-09-18T01:32:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 314384384 bytes 2024-09-18T01:32:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 374725632 bytes 2024-09-18T01:32:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 435478528 bytes 2024-09-18T01:32:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 493668352 bytes 2024-09-18T01:32:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 543089664 bytes 2024-09-18T01:32:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 612520960 bytes 2024-09-18T01:32:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 645459968 bytes 2024-09-18T01:32:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 684492800 bytes 2024-09-18T01:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 778475520 bytes 2024-09-18T01:32:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 852044800 bytes 2024-09-18T01:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 916312064 bytes 2024-09-18T01:32:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 958713856 bytes 2024-09-18T01:32:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 976087040 bytes 2024-09-18T01:32:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 1045436416 bytes 2024-09-18T01:32:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1046046720 bytes 2024-09-18T01:32:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1092135936 bytes 2024-09-18T01:32:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1121996800 bytes 2024-09-18T01:32:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1145728000 bytes 2024-09-18T01:32:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1168733184 bytes 2024-09-18T01:32:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1236561920 bytes 2024-09-18T01:32:57.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1287949312 bytes 2024-09-18T01:32:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1352561664 bytes 2024-09-18T01:32:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1388474368 bytes 2024-09-18T01:32:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1409476608 bytes 2024-09-18T01:32:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1497182208 bytes 2024-09-18T01:32:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1541440512 bytes 2024-09-18T01:32:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1547615232 bytes 2024-09-18T01:32:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1581647872 bytes 2024-09-18T01:32:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1629432832 bytes 2024-09-18T01:32:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1658713088 bytes 2024-09-18T01:32:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1684254720 bytes 2024-09-18T01:32:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1700381696 bytes 2024-09-18T01:32:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1732396032 bytes 2024-09-18T01:32:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1784266752 bytes 2024-09-18T01:32:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1806526464 bytes 2024-09-18T01:32:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1827516416 bytes 2024-09-18T01:32:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1901575168 bytes 2024-09-18T01:32:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1952977920 bytes 2024-09-18T01:32:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2014217216 bytes 2024-09-18T01:33:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2014812160 bytes 2024-09-18T01:33:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2070249472 bytes 2024-09-18T01:33:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:33:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:33:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:33:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:33:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_31 2024-09-18T01:33:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:33:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-18T01:33:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:33:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:33:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:33:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:33:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2024-09-18T01:33:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:33:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:33:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-18T01:33:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:33:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-18T01:33:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:33:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-18T01:33:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_31': 2024-09-18T01:33:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:33:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:33:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:33:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13e957bfad43 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13e957bfad43 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:00 2024 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:00 2024 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:00 2024 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 5d13ee44-9f2d-44a2-bfb5-f6f0c940027e 2024-09-18T01:33:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:33:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:33:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-18T01:33:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:33:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:33:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=12 2024-09-18T01:33:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:33:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:33:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 37442560 bytes 2024-09-18T01:33:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 83338240 bytes 2024-09-18T01:33:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 218048512 bytes 2024-09-18T01:33:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 259904512 bytes 2024-09-18T01:33:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 283603968 bytes 2024-09-18T01:33:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 352497664 bytes 2024-09-18T01:33:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 362702848 bytes 2024-09-18T01:33:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 363332608 bytes 2024-09-18T01:33:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 418824192 bytes 2024-09-18T01:33:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 507916288 bytes 2024-09-18T01:33:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 531468288 bytes 2024-09-18T01:33:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 622016512 bytes 2024-09-18T01:33:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 749418496 bytes 2024-09-18T01:33:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 794898432 bytes 2024-09-18T01:33:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 833312768 bytes 2024-09-18T01:33:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 857605120 bytes 2024-09-18T01:33:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 877240320 bytes 2024-09-18T01:33:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 923461632 bytes 2024-09-18T01:33:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 972960768 bytes 2024-09-18T01:33:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1017716736 bytes 2024-09-18T01:33:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1061701632 bytes 2024-09-18T01:33:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1092239360 bytes 2024-09-18T01:33:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1092853760 bytes 2024-09-18T01:33:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1153171456 bytes 2024-09-18T01:33:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1162037248 bytes 2024-09-18T01:33:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1253816320 bytes 2024-09-18T01:33:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1329153024 bytes 2024-09-18T01:33:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1403342848 bytes 2024-09-18T01:33:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1450029056 bytes 2024-09-18T01:33:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1545178112 bytes 2024-09-18T01:33:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1578378240 bytes 2024-09-18T01:33:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1619538944 bytes 2024-09-18T01:33:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1651734528 bytes 2024-09-18T01:33:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1682814976 bytes 2024-09-18T01:33:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1729897472 bytes 2024-09-18T01:33:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1767503872 bytes 2024-09-18T01:33:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1794259968 bytes 2024-09-18T01:33:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1857092608 bytes 2024-09-18T01:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1889512448 bytes 2024-09-18T01:33:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1890145280 bytes 2024-09-18T01:33:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1975310336 bytes 2024-09-18T01:33:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2018487296 bytes 2024-09-18T01:33:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2022026240 bytes 2024-09-18T01:33:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2050981888 bytes 2024-09-18T01:33:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2099591168 bytes 2024-09-18T01:33:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2100219904 bytes 2024-09-18T01:33:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2159348736 bytes 2024-09-18T01:33:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2207592448 bytes 2024-09-18T01:33:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2237039616 bytes 2024-09-18T01:33:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2279344128 bytes 2024-09-18T01:33:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2294813696 bytes 2024-09-18T01:33:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2327171072 bytes 2024-09-18T01:33:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2378151936 bytes 2024-09-18T01:33:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2421925888 bytes 2024-09-18T01:33:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2422712320 bytes 2024-09-18T01:33:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2501098496 bytes 2024-09-18T01:33:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2524364800 bytes 2024-09-18T01:33:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2551671808 bytes 2024-09-18T01:33:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2587200512 bytes 2024-09-18T01:33:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2644541440 bytes 2024-09-18T01:33:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2719298560 bytes 2024-09-18T01:33:08.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2771326976 bytes 2024-09-18T01:33:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2812582912 bytes 2024-09-18T01:33:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2834659328 bytes 2024-09-18T01:33:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2855003136 bytes 2024-09-18T01:33:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2962367488 bytes 2024-09-18T01:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3035786240 bytes 2024-09-18T01:33:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3064069120 bytes 2024-09-18T01:33:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3064659968 bytes 2024-09-18T01:33:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3128875008 bytes 2024-09-18T01:33:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3178893312 bytes 2024-09-18T01:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3182544896 bytes 2024-09-18T01:33:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3195759616 bytes 2024-09-18T01:33:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3210824704 bytes 2024-09-18T01:33:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3282014208 bytes 2024-09-18T01:33:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3301978112 bytes 2024-09-18T01:33:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3335221248 bytes 2024-09-18T01:33:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3445569536 bytes 2024-09-18T01:33:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3481369600 bytes 2024-09-18T01:33:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3518842880 bytes 2024-09-18T01:33:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3570660352 bytes 2024-09-18T01:33:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3588799488 bytes 2024-09-18T01:33:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3620455424 bytes 2024-09-18T01:33:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3659812864 bytes 2024-09-18T01:33:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3743484928 bytes 2024-09-18T01:33:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3798989824 bytes 2024-09-18T01:33:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3845740544 bytes 2024-09-18T01:33:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3895093248 bytes 2024-09-18T01:33:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3934775296 bytes 2024-09-18T01:33:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 4016490496 bytes 2024-09-18T01:33:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4090670080 bytes 2024-09-18T01:33:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4125614080 bytes 2024-09-18T01:33:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4161027072 bytes 2024-09-18T01:33:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4251131904 bytes 2024-09-18T01:33:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4301300736 bytes 2024-09-18T01:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_32 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:33:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:33:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:33:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:33:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:33:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-18T01:33:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:33:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-18T01:33:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_32': 2024-09-18T01:33:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:33:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:33:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 13f51bad6923 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.13f51bad6923 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:13 2024 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:13 2024 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:13 2024 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: b485039f-a047-4ba3-86d2-a7ddfa23cdc8 2024-09-18T01:33:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:33:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:33:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-18T01:33:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:33:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:33:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:33:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:33:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:33:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 71693312 bytes 2024-09-18T01:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 93147136 bytes 2024-09-18T01:33:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 141696000 bytes 2024-09-18T01:33:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 195684352 bytes 2024-09-18T01:33:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 251492352 bytes 2024-09-18T01:33:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 271878144 bytes 2024-09-18T01:33:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 340482048 bytes 2024-09-18T01:33:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 399722496 bytes 2024-09-18T01:33:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 428589056 bytes 2024-09-18T01:33:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 443316224 bytes 2024-09-18T01:33:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 486228992 bytes 2024-09-18T01:33:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 525680640 bytes 2024-09-18T01:33:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 667713536 bytes 2024-09-18T01:33:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 693296128 bytes 2024-09-18T01:33:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 731294720 bytes 2024-09-18T01:33:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 817849344 bytes 2024-09-18T01:33:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 818475008 bytes 2024-09-18T01:33:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 846444544 bytes 2024-09-18T01:33:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 992344064 bytes 2024-09-18T01:33:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1007502336 bytes 2024-09-18T01:33:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1054627840 bytes 2024-09-18T01:33:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1067862016 bytes 2024-09-18T01:33:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1144126464 bytes 2024-09-18T01:33:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1188098048 bytes 2024-09-18T01:33:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1236508672 bytes 2024-09-18T01:33:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1256275968 bytes 2024-09-18T01:33:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1321183232 bytes 2024-09-18T01:33:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1329690624 bytes 2024-09-18T01:33:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1344175104 bytes 2024-09-18T01:33:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1413639168 bytes 2024-09-18T01:33:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1493278720 bytes 2024-09-18T01:33:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1564315648 bytes 2024-09-18T01:33:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1599425536 bytes 2024-09-18T01:33:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1670343680 bytes 2024-09-18T01:33:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1688119296 bytes 2024-09-18T01:33:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1731691520 bytes 2024-09-18T01:33:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1778181120 bytes 2024-09-18T01:33:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1807592448 bytes 2024-09-18T01:33:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1944770560 bytes 2024-09-18T01:33:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 2010148864 bytes 2024-09-18T01:33:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 2031419392 bytes 2024-09-18T01:33:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2053382144 bytes 2024-09-18T01:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2078258176 bytes 2024-09-18T01:33:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2111632384 bytes 2024-09-18T01:33:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2196000768 bytes 2024-09-18T01:33:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2229810176 bytes 2024-09-18T01:33:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2262361088 bytes 2024-09-18T01:33:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2320763904 bytes 2024-09-18T01:33:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2354612224 bytes 2024-09-18T01:33:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2401069056 bytes 2024-09-18T01:33:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2460079104 bytes 2024-09-18T01:33:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2478976000 bytes 2024-09-18T01:33:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2520581120 bytes 2024-09-18T01:33:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2543883264 bytes 2024-09-18T01:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2561746944 bytes 2024-09-18T01:33:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2596249600 bytes 2024-09-18T01:33:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2596873216 bytes 2024-09-18T01:33:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2597493760 bytes 2024-09-18T01:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2603206656 bytes 2024-09-18T01:33:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2682409984 bytes 2024-09-18T01:33:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2683025408 bytes 2024-09-18T01:33:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2742581248 bytes 2024-09-18T01:33:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2774283264 bytes 2024-09-18T01:33:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2791224320 bytes 2024-09-18T01:33:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2847096832 bytes 2024-09-18T01:33:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2882360320 bytes 2024-09-18T01:33:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2929199104 bytes 2024-09-18T01:33:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2942112768 bytes 2024-09-18T01:33:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2969653248 bytes 2024-09-18T01:33:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2998087680 bytes 2024-09-18T01:33:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3029350400 bytes 2024-09-18T01:33:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3065986048 bytes 2024-09-18T01:33:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_33 2024-09-18T01:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-18T01:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:33:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:33:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:33:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:33:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2024-09-18T01:33:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-18T01:33:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:33:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-18T01:33:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_33': 2024-09-18T01:33:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 1401997f213b 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.1401997f213b 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:22 2024 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:22 2024 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:22 2024 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:33:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:33:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: a3efecd5-88da-4f50-a556-abf649ac4e9a 2024-09-18T01:33:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:33:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:33:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-18T01:33:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:33:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:33:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=13 2024-09-18T01:33:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:33:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:33:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 57284608 bytes 2024-09-18T01:33:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 69734400 bytes 2024-09-18T01:33:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 108866560 bytes 2024-09-18T01:33:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 131318784 bytes 2024-09-18T01:33:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 177528832 bytes 2024-09-18T01:33:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 200356864 bytes 2024-09-18T01:33:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 226202624 bytes 2024-09-18T01:33:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 247515136 bytes 2024-09-18T01:33:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 326306816 bytes 2024-09-18T01:33:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 347136000 bytes 2024-09-18T01:33:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 391473152 bytes 2024-09-18T01:33:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 442779648 bytes 2024-09-18T01:33:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 464187392 bytes 2024-09-18T01:33:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 487255040 bytes 2024-09-18T01:33:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 507726848 bytes 2024-09-18T01:33:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 515918848 bytes 2024-09-18T01:33:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 575225856 bytes 2024-09-18T01:33:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 632876032 bytes 2024-09-18T01:33:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 712375296 bytes 2024-09-18T01:33:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 746135552 bytes 2024-09-18T01:33:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 782707712 bytes 2024-09-18T01:33:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 822189056 bytes 2024-09-18T01:33:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 893321216 bytes 2024-09-18T01:33:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 940081152 bytes 2024-09-18T01:33:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 959752192 bytes 2024-09-18T01:33:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 981184512 bytes 2024-09-18T01:33:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 985919488 bytes 2024-09-18T01:33:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1034676224 bytes 2024-09-18T01:33:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1038027776 bytes 2024-09-18T01:33:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1063690240 bytes 2024-09-18T01:33:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1118081024 bytes 2024-09-18T01:33:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1118678016 bytes 2024-09-18T01:33:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1127218176 bytes 2024-09-18T01:33:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1175403520 bytes 2024-09-18T01:33:27.937 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3738812910811362) > weight (0.25) 2024-09-18T01:33:27.937 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5335884925673823) > weight (0.25) 2024-09-18T01:33:27.937 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7326210448154815) > weight (0.25) 2024-09-18T01:33:27.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3068893755208153) > weight (0.25) 2024-09-18T01:33:27.938 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-18T01:33:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1197281280 bytes 2024-09-18T01:33:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1240003584 bytes 2024-09-18T01:33:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1256107008 bytes 2024-09-18T01:33:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1348306944 bytes 2024-09-18T01:33:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1368957952 bytes 2024-09-18T01:33:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1387719680 bytes 2024-09-18T01:33:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1444766720 bytes 2024-09-18T01:33:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1455472640 bytes 2024-09-18T01:33:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1475240960 bytes 2024-09-18T01:33:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1534579712 bytes 2024-09-18T01:33:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1605393408 bytes 2024-09-18T01:33:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1634778112 bytes 2024-09-18T01:33:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1657742336 bytes 2024-09-18T01:33:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1671858176 bytes 2024-09-18T01:33:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1722516480 bytes 2024-09-18T01:33:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1738708992 bytes 2024-09-18T01:33:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1750019072 bytes 2024-09-18T01:33:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1801528320 bytes 2024-09-18T01:33:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 1825207296 bytes 2024-09-18T01:33:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 1843964928 bytes 2024-09-18T01:33:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 1928512512 bytes 2024-09-18T01:33:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 1929124864 bytes 2024-09-18T01:33:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 1965452288 bytes 2024-09-18T01:33:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 1966071808 bytes 2024-09-18T01:33:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 1966692352 bytes 2024-09-18T01:33:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2024944640 bytes 2024-09-18T01:33:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2061705216 bytes 2024-09-18T01:33:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2098538496 bytes 2024-09-18T01:33:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2142496768 bytes 2024-09-18T01:33:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2143100928 bytes 2024-09-18T01:33:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2165613568 bytes 2024-09-18T01:33:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2188369920 bytes 2024-09-18T01:33:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2207805440 bytes 2024-09-18T01:33:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2232698880 bytes 2024-09-18T01:33:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2258946048 bytes 2024-09-18T01:33:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2280346624 bytes 2024-09-18T01:33:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2311130112 bytes 2024-09-18T01:33:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2355022848 bytes 2024-09-18T01:33:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2401672192 bytes 2024-09-18T01:33:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 2445047808 bytes 2024-09-18T01:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 2476638208 bytes 2024-09-18T01:33:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 2531877888 bytes 2024-09-18T01:33:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 2573351936 bytes 2024-09-18T01:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 2650049536 bytes 2024-09-18T01:33:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 2671285248 bytes 2024-09-18T01:33:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 2686667776 bytes 2024-09-18T01:33:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 2764295168 bytes 2024-09-18T01:33:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 2768847872 bytes 2024-09-18T01:33:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 2833169408 bytes 2024-09-18T01:33:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 2848521216 bytes 2024-09-18T01:33:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 2862298112 bytes 2024-09-18T01:33:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 2904373248 bytes 2024-09-18T01:33:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3053104128 bytes 2024-09-18T01:33:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3092882432 bytes 2024-09-18T01:33:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3122670592 bytes 2024-09-18T01:33:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3148022784 bytes 2024-09-18T01:33:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3173197824 bytes 2024-09-18T01:33:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3278618624 bytes 2024-09-18T01:33:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 3320629248 bytes 2024-09-18T01:33:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 3363934208 bytes 2024-09-18T01:33:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 3422502912 bytes 2024-09-18T01:33:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 3447511040 bytes 2024-09-18T01:33:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 3472899072 bytes 2024-09-18T01:33:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 3494492160 bytes 2024-09-18T01:33:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 3544235008 bytes 2024-09-18T01:33:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 3587590144 bytes 2024-09-18T01:33:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 3655226368 bytes 2024-09-18T01:33:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 3666974720 bytes 2024-09-18T01:33:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 3742689280 bytes 2024-09-18T01:33:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 3805484032 bytes 2024-09-18T01:33:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 3811231744 bytes 2024-09-18T01:33:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 3821137920 bytes 2024-09-18T01:33:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 3872162816 bytes 2024-09-18T01:33:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 3872765952 bytes 2024-09-18T01:33:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 3904330752 bytes 2024-09-18T01:33:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_34 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:33:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-18T01:33:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:33:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:33:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:33:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:33:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-18T01:33:37.728 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-18T01:33:37.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-18T01:33:37.729 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-18T01:33:37.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:33:37.729+0000 7f34498c1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 44913) UID: 1000 2024-09-18T01:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:33:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-18T01:33:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_34': 2024-09-18T01:33:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:33:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 140d75c8c5ef 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.140d75c8c5ef 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:36 2024 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:36 2024 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:36 2024 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 5d3ed84a-3952-450d-a72a-a0fbf5938579 2024-09-18T01:33:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:33:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:33:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-18T01:33:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:33:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:33:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=12 2024-09-18T01:33:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:33:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:33:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 80694272 bytes 2024-09-18T01:33:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 120329216 bytes 2024-09-18T01:33:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 123445248 bytes 2024-09-18T01:33:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 199771136 bytes 2024-09-18T01:33:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 253977600 bytes 2024-09-18T01:33:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 306210816 bytes 2024-09-18T01:33:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 346810368 bytes 2024-09-18T01:33:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 368431104 bytes 2024-09-18T01:33:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 379409408 bytes 2024-09-18T01:33:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 384477184 bytes 2024-09-18T01:33:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 427601920 bytes 2024-09-18T01:33:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 526355456 bytes 2024-09-18T01:33:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 574241792 bytes 2024-09-18T01:33:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 649108480 bytes 2024-09-18T01:33:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 708750336 bytes 2024-09-18T01:33:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 743375872 bytes 2024-09-18T01:33:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 796100608 bytes 2024-09-18T01:33:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 841913344 bytes 2024-09-18T01:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 880679936 bytes 2024-09-18T01:33:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 888922112 bytes 2024-09-18T01:33:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 929924096 bytes 2024-09-18T01:33:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 972353536 bytes 2024-09-18T01:33:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1015971840 bytes 2024-09-18T01:33:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1059497984 bytes 2024-09-18T01:33:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1069118464 bytes 2024-09-18T01:33:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1110194176 bytes 2024-09-18T01:33:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1131042816 bytes 2024-09-18T01:33:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1212849152 bytes 2024-09-18T01:33:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1279474688 bytes 2024-09-18T01:33:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1398216704 bytes 2024-09-18T01:33:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1409333248 bytes 2024-09-18T01:33:42.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1483160576 bytes 2024-09-18T01:33:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1483789312 bytes 2024-09-18T01:33:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1528011776 bytes 2024-09-18T01:33:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1590994944 bytes 2024-09-18T01:33:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1597808640 bytes 2024-09-18T01:33:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1666188288 bytes 2024-09-18T01:33:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1686678528 bytes 2024-09-18T01:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1700161536 bytes 2024-09-18T01:33:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1718939648 bytes 2024-09-18T01:33:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1792446464 bytes 2024-09-18T01:33:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1851815936 bytes 2024-09-18T01:33:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1928751104 bytes 2024-09-18T01:33:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1992812544 bytes 2024-09-18T01:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2033928192 bytes 2024-09-18T01:33:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2044780544 bytes 2024-09-18T01:33:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2107865088 bytes 2024-09-18T01:33:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2170947584 bytes 2024-09-18T01:33:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2230516736 bytes 2024-09-18T01:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2272792576 bytes 2024-09-18T01:33:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2332289024 bytes 2024-09-18T01:33:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2383075328 bytes 2024-09-18T01:33:44.730 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-18T01:33:44.730 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:33:44.730 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-18T01:33:44.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-18T01:33:44.730 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-18T01:33:44.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-18T01:33:44.733 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-18T01:33:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2432506880 bytes 2024-09-18T01:33:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2444946432 bytes 2024-09-18T01:33:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2542936064 bytes 2024-09-18T01:33:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2565125120 bytes 2024-09-18T01:33:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2580568064 bytes 2024-09-18T01:33:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2584646656 bytes 2024-09-18T01:33:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2624936960 bytes 2024-09-18T01:33:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2631025664 bytes 2024-09-18T01:33:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2742401024 bytes 2024-09-18T01:33:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2780357632 bytes 2024-09-18T01:33:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2835062784 bytes 2024-09-18T01:33:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2875486208 bytes 2024-09-18T01:33:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2895164416 bytes 2024-09-18T01:33:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2943219712 bytes 2024-09-18T01:33:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3031541760 bytes 2024-09-18T01:33:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3072303104 bytes 2024-09-18T01:33:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3088626688 bytes 2024-09-18T01:33:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3108366336 bytes 2024-09-18T01:33:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3114489856 bytes 2024-09-18T01:33:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3204411392 bytes 2024-09-18T01:33:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3270872064 bytes 2024-09-18T01:33:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3304735744 bytes 2024-09-18T01:33:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3317556224 bytes 2024-09-18T01:33:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3361718272 bytes 2024-09-18T01:33:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3373958144 bytes 2024-09-18T01:33:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3469074432 bytes 2024-09-18T01:33:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3526170624 bytes 2024-09-18T01:33:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3577032704 bytes 2024-09-18T01:33:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3618547712 bytes 2024-09-18T01:33:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3639562240 bytes 2024-09-18T01:33:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3677511680 bytes 2024-09-18T01:33:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3683111936 bytes 2024-09-18T01:33:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3763826688 bytes 2024-09-18T01:33:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3791056896 bytes 2024-09-18T01:33:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3814405120 bytes 2024-09-18T01:33:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3856534528 bytes 2024-09-18T01:33:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3922307072 bytes 2024-09-18T01:33:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3973021696 bytes 2024-09-18T01:33:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3978820608 bytes 2024-09-18T01:33:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4058614784 bytes 2024-09-18T01:33:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:33:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:33:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:33:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:33:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_35 2024-09-18T01:33:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-18T01:33:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:33:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:33:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-18T01:33:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:33:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-18T01:33:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_35': 2024-09-18T01:33:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 141f43c43e69 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.141f43c43e69 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:49 2024 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:49 2024 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:49 2024 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:33:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:33:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 1650cda4-929a-407a-b124-0d014c9e4fa3 2024-09-18T01:33:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:33:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:33:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-18T01:33:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:33:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:33:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:33:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:33:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:33:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 43609088 bytes 2024-09-18T01:33:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 44216320 bytes 2024-09-18T01:33:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 44856320 bytes 2024-09-18T01:33:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 65805312 bytes 2024-09-18T01:33:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 136195072 bytes 2024-09-18T01:33:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 172054528 bytes 2024-09-18T01:33:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 211639296 bytes 2024-09-18T01:33:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 316204032 bytes 2024-09-18T01:33:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 360897536 bytes 2024-09-18T01:33:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 406246400 bytes 2024-09-18T01:33:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 428560384 bytes 2024-09-18T01:33:51.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 453379072 bytes 2024-09-18T01:33:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 487237632 bytes 2024-09-18T01:33:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 511866880 bytes 2024-09-18T01:33:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 561163264 bytes 2024-09-18T01:33:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 581542912 bytes 2024-09-18T01:33:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 687170560 bytes 2024-09-18T01:33:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 689380352 bytes 2024-09-18T01:33:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 741708800 bytes 2024-09-18T01:33:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 796983296 bytes 2024-09-18T01:33:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 866833408 bytes 2024-09-18T01:33:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 871042048 bytes 2024-09-18T01:33:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 894527488 bytes 2024-09-18T01:33:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 952896512 bytes 2024-09-18T01:33:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 999012352 bytes 2024-09-18T01:33:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1030203392 bytes 2024-09-18T01:33:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1092874240 bytes 2024-09-18T01:33:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1109022720 bytes 2024-09-18T01:33:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1115849728 bytes 2024-09-18T01:33:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1124483072 bytes 2024-09-18T01:33:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1177903104 bytes 2024-09-18T01:33:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1186415616 bytes 2024-09-18T01:33:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1230854144 bytes 2024-09-18T01:33:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1248517120 bytes 2024-09-18T01:33:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1317515264 bytes 2024-09-18T01:33:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1394750464 bytes 2024-09-18T01:33:54.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1403432960 bytes 2024-09-18T01:33:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1464899584 bytes 2024-09-18T01:33:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1494219776 bytes 2024-09-18T01:33:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1521499136 bytes 2024-09-18T01:33:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1533702144 bytes 2024-09-18T01:33:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1576757248 bytes 2024-09-18T01:33:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1644860416 bytes 2024-09-18T01:33:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1674846208 bytes 2024-09-18T01:33:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1742599168 bytes 2024-09-18T01:33:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1747274752 bytes 2024-09-18T01:33:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1795487744 bytes 2024-09-18T01:33:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1814111232 bytes 2024-09-18T01:33:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1814743040 bytes 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_36 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2024-09-18T01:33:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:33:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:33:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:33:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-18T01:33:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:33:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:33:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-18T01:33:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:33:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-18T01:33:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_36': 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 142be19c29a9 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.142be19c29a9 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:56 2024 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:56 2024 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:56 2024 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:33:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 8455fc5d-6748-424e-ae66-ca58df099087 2024-09-18T01:33:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:33:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:33:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-18T01:33:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:33:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:33:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:33:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:33:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=8 2024-09-18T01:33:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:33:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:33:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 622592 bytes 2024-09-18T01:33:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 27409408 bytes 2024-09-18T01:33:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 34078720 bytes 2024-09-18T01:33:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 58381312 bytes 2024-09-18T01:33:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 89610240 bytes 2024-09-18T01:33:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 151947264 bytes 2024-09-18T01:33:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 182408192 bytes 2024-09-18T01:33:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 233253888 bytes 2024-09-18T01:33:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 260845568 bytes 2024-09-18T01:33:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 301928448 bytes 2024-09-18T01:33:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 356850688 bytes 2024-09-18T01:33:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 387892224 bytes 2024-09-18T01:33:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 398283776 bytes 2024-09-18T01:33:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 424977408 bytes 2024-09-18T01:33:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 440179712 bytes 2024-09-18T01:33:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 450512896 bytes 2024-09-18T01:33:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 485467136 bytes 2024-09-18T01:33:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 564552704 bytes 2024-09-18T01:33:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 611682304 bytes 2024-09-18T01:33:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 641345536 bytes 2024-09-18T01:33:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 651801600 bytes 2024-09-18T01:33:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 696394752 bytes 2024-09-18T01:33:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 772381696 bytes 2024-09-18T01:34:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 822005760 bytes 2024-09-18T01:34:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 838562816 bytes 2024-09-18T01:34:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 858067968 bytes 2024-09-18T01:34:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 887825408 bytes 2024-09-18T01:34:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 889700352 bytes 2024-09-18T01:34:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 956675072 bytes 2024-09-18T01:34:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 982834176 bytes 2024-09-18T01:34:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1042984960 bytes 2024-09-18T01:34:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1085188096 bytes 2024-09-18T01:34:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1138909184 bytes 2024-09-18T01:34:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1189480448 bytes 2024-09-18T01:34:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1199224832 bytes 2024-09-18T01:34:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1239314432 bytes 2024-09-18T01:34:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1254591488 bytes 2024-09-18T01:34:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1327905792 bytes 2024-09-18T01:34:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1412914176 bytes 2024-09-18T01:34:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1519989760 bytes 2024-09-18T01:34:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1585713152 bytes 2024-09-18T01:34:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1614688256 bytes 2024-09-18T01:34:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1615312896 bytes 2024-09-18T01:34:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1652041728 bytes 2024-09-18T01:34:02.588 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1662288896 bytes 2024-09-18T01:34:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1686565888 bytes 2024-09-18T01:34:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1758957568 bytes 2024-09-18T01:34:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1817622528 bytes 2024-09-18T01:34:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1843873792 bytes 2024-09-18T01:34:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1897821184 bytes 2024-09-18T01:34:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1915730944 bytes 2024-09-18T01:34:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1969620992 bytes 2024-09-18T01:34:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 1986916352 bytes 2024-09-18T01:34:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2051011584 bytes 2024-09-18T01:34:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2150203392 bytes 2024-09-18T01:34:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2269961216 bytes 2024-09-18T01:34:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2281022464 bytes 2024-09-18T01:34:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2317748224 bytes 2024-09-18T01:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2409582592 bytes 2024-09-18T01:34:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2418697216 bytes 2024-09-18T01:34:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2451906560 bytes 2024-09-18T01:34:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2492376064 bytes 2024-09-18T01:34:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2514608128 bytes 2024-09-18T01:34:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2616452096 bytes 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_37 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2024-09-18T01:34:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:34:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:34:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-18T01:34:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:34:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-18T01:34:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:34:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_37': 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 14371dccbcce 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.14371dccbcce 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:05 2024 2024-09-18T01:34:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:05 2024 2024-09-18T01:34:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:05 2024 2024-09-18T01:34:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:34:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:34:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 45c73ce1-e604-4921-a91c-891688be3288 2024-09-18T01:34:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:34:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:34:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-18T01:34:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:34:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:34:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:34:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:34:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:34:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 32865280 bytes 2024-09-18T01:34:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 71372800 bytes 2024-09-18T01:34:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 127258624 bytes 2024-09-18T01:34:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 181353472 bytes 2024-09-18T01:34:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 271933440 bytes 2024-09-18T01:34:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 317025280 bytes 2024-09-18T01:34:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 364424192 bytes 2024-09-18T01:34:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 485012480 bytes 2024-09-18T01:34:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 502788096 bytes 2024-09-18T01:34:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 530661376 bytes 2024-09-18T01:34:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 533405696 bytes 2024-09-18T01:34:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 558704640 bytes 2024-09-18T01:34:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 592537600 bytes 2024-09-18T01:34:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 652614656 bytes 2024-09-18T01:34:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 734911488 bytes 2024-09-18T01:34:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 784667648 bytes 2024-09-18T01:34:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 807784448 bytes 2024-09-18T01:34:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 819865600 bytes 2024-09-18T01:34:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 873445376 bytes 2024-09-18T01:34:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 954635264 bytes 2024-09-18T01:34:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 987412480 bytes 2024-09-18T01:34:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1014844416 bytes 2024-09-18T01:34:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1069408256 bytes 2024-09-18T01:34:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1080086528 bytes 2024-09-18T01:34:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1119124480 bytes 2024-09-18T01:34:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1203892224 bytes 2024-09-18T01:34:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1204489216 bytes 2024-09-18T01:34:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1258077184 bytes 2024-09-18T01:34:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1318871040 bytes 2024-09-18T01:34:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1356248064 bytes 2024-09-18T01:34:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1380963328 bytes 2024-09-18T01:34:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1402305536 bytes 2024-09-18T01:34:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1462974464 bytes 2024-09-18T01:34:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1463595008 bytes 2024-09-18T01:34:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1514276864 bytes 2024-09-18T01:34:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1588866048 bytes 2024-09-18T01:34:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1675418624 bytes 2024-09-18T01:34:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1703322624 bytes 2024-09-18T01:34:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1733654528 bytes 2024-09-18T01:34:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:34:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:34:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:34:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:34:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_38 2024-09-18T01:34:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-18T01:34:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:34:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:34:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:34:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:34:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2024-09-18T01:34:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-18T01:34:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:34:12.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-18T01:34:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_38': 2024-09-18T01:34:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:34:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:34:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:34:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 14438b1e88ee 2024-09-18T01:34:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.14438b1e88ee 2024-09-18T01:34:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:34:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:34:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:34:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:34:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:11 2024 2024-09-18T01:34:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:11 2024 2024-09-18T01:34:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:11 2024 2024-09-18T01:34:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:34:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:34:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: db636ff1-46c8-441b-8a47-bca8ce4f7f7e 2024-09-18T01:34:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:34:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:34:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-18T01:34:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:34:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:34:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:34:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:34:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:34:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 56509440 bytes 2024-09-18T01:34:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 103270400 bytes 2024-09-18T01:34:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 134213632 bytes 2024-09-18T01:34:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 160501760 bytes 2024-09-18T01:34:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 227655680 bytes 2024-09-18T01:34:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 284624896 bytes 2024-09-18T01:34:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 316626944 bytes 2024-09-18T01:34:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 382240768 bytes 2024-09-18T01:34:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 391109632 bytes 2024-09-18T01:34:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 433529856 bytes 2024-09-18T01:34:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 488226816 bytes 2024-09-18T01:34:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 498692096 bytes 2024-09-18T01:34:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 557804544 bytes 2024-09-18T01:34:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 606126080 bytes 2024-09-18T01:34:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 630099968 bytes 2024-09-18T01:34:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 675999744 bytes 2024-09-18T01:34:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 730181632 bytes 2024-09-18T01:34:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 754969600 bytes 2024-09-18T01:34:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 817419264 bytes 2024-09-18T01:34:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 866549760 bytes 2024-09-18T01:34:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 894567424 bytes 2024-09-18T01:34:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 923236352 bytes 2024-09-18T01:34:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 930073600 bytes 2024-09-18T01:34:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 930696192 bytes 2024-09-18T01:34:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 969777152 bytes 2024-09-18T01:34:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1042410496 bytes 2024-09-18T01:34:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1053614080 bytes 2024-09-18T01:34:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1098699776 bytes 2024-09-18T01:34:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1172016128 bytes 2024-09-18T01:34:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1212174336 bytes 2024-09-18T01:34:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1248592896 bytes 2024-09-18T01:34:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1277198336 bytes 2024-09-18T01:34:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1317492736 bytes 2024-09-18T01:34:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1318106112 bytes 2024-09-18T01:34:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1376643072 bytes 2024-09-18T01:34:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1422348288 bytes 2024-09-18T01:34:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1508549632 bytes 2024-09-18T01:34:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1553483776 bytes 2024-09-18T01:34:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1577965568 bytes 2024-09-18T01:34:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1644282880 bytes 2024-09-18T01:34:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1682804736 bytes 2024-09-18T01:34:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1691383808 bytes 2024-09-18T01:34:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1773760512 bytes 2024-09-18T01:34:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1820346368 bytes 2024-09-18T01:34:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1881686016 bytes 2024-09-18T01:34:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1935698944 bytes 2024-09-18T01:34:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1997918208 bytes 2024-09-18T01:34:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2074705920 bytes 2024-09-18T01:34:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2110142464 bytes 2024-09-18T01:34:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2115170304 bytes 2024-09-18T01:34:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2192130048 bytes 2024-09-18T01:34:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2208220160 bytes 2024-09-18T01:34:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2217194496 bytes 2024-09-18T01:34:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2231414784 bytes 2024-09-18T01:34:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2297186304 bytes 2024-09-18T01:34:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2400025600 bytes 2024-09-18T01:34:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2491439104 bytes 2024-09-18T01:34:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2532443136 bytes 2024-09-18T01:34:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2543168512 bytes 2024-09-18T01:34:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2585976832 bytes 2024-09-18T01:34:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2603322368 bytes 2024-09-18T01:34:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2655963136 bytes 2024-09-18T01:34:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2686123008 bytes 2024-09-18T01:34:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2733167616 bytes 2024-09-18T01:34:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2772826112 bytes 2024-09-18T01:34:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2815724544 bytes 2024-09-18T01:34:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2869997568 bytes 2024-09-18T01:34:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2899589120 bytes 2024-09-18T01:34:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2912203776 bytes 2024-09-18T01:34:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:34:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:34:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:34:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:34:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_39 2024-09-18T01:34:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-18T01:34:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:34:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:34:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-18T01:34:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:34:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:34:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:34:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:34:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-18T01:34:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:34:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_39': 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 144fd8c468d0 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.144fd8c468d0 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:34:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:34:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:34:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:21 2024 2024-09-18T01:34:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:21 2024 2024-09-18T01:34:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:21 2024 2024-09-18T01:34:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:34:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:34:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 900fab41-f944-4927-8c53-bac7011186e3 2024-09-18T01:34:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-18T01:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:34:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=11 2024-09-18T01:34:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:34:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:34:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 41365504 bytes 2024-09-18T01:34:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 161746944 bytes 2024-09-18T01:34:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 195360768 bytes 2024-09-18T01:34:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 204116992 bytes 2024-09-18T01:34:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 234387456 bytes 2024-09-18T01:34:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 302715904 bytes 2024-09-18T01:34:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 328098816 bytes 2024-09-18T01:34:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 370899968 bytes 2024-09-18T01:34:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 436835328 bytes 2024-09-18T01:34:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 454006784 bytes 2024-09-18T01:34:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 504363008 bytes 2024-09-18T01:34:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 504971264 bytes 2024-09-18T01:34:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 511894528 bytes 2024-09-18T01:34:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 512518144 bytes 2024-09-18T01:34:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 526550016 bytes 2024-09-18T01:34:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 527192064 bytes 2024-09-18T01:34:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 571468800 bytes 2024-09-18T01:34:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 662478848 bytes 2024-09-18T01:34:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 663084032 bytes 2024-09-18T01:34:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 706079744 bytes 2024-09-18T01:34:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 757857280 bytes 2024-09-18T01:34:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 789695488 bytes 2024-09-18T01:34:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 808727552 bytes 2024-09-18T01:34:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 834264064 bytes 2024-09-18T01:34:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 897210368 bytes 2024-09-18T01:34:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 939967488 bytes 2024-09-18T01:34:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1005867008 bytes 2024-09-18T01:34:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1068033024 bytes 2024-09-18T01:34:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1133114368 bytes 2024-09-18T01:34:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1159997440 bytes 2024-09-18T01:34:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1203179520 bytes 2024-09-18T01:34:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1204415488 bytes 2024-09-18T01:34:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1221870592 bytes 2024-09-18T01:34:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1250334720 bytes 2024-09-18T01:34:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1300604928 bytes 2024-09-18T01:34:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1301233664 bytes 2024-09-18T01:34:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1325519872 bytes 2024-09-18T01:34:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1360881664 bytes 2024-09-18T01:34:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1419961344 bytes 2024-09-18T01:34:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1514630144 bytes 2024-09-18T01:34:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1607968768 bytes 2024-09-18T01:34:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1650290688 bytes 2024-09-18T01:34:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1650897920 bytes 2024-09-18T01:34:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1673769984 bytes 2024-09-18T01:34:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1695166464 bytes 2024-09-18T01:34:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1733751808 bytes 2024-09-18T01:34:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1800519680 bytes 2024-09-18T01:34:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1801138176 bytes 2024-09-18T01:34:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1860642816 bytes 2024-09-18T01:34:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1923730432 bytes 2024-09-18T01:34:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1956955136 bytes 2024-09-18T01:34:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1979297792 bytes 2024-09-18T01:34:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2094511104 bytes 2024-09-18T01:34:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2137206784 bytes 2024-09-18T01:34:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2171596800 bytes 2024-09-18T01:34:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2186851328 bytes 2024-09-18T01:34:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2215777280 bytes 2024-09-18T01:34:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2244732928 bytes 2024-09-18T01:34:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2302420992 bytes 2024-09-18T01:34:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2322098176 bytes 2024-09-18T01:34:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2373291008 bytes 2024-09-18T01:34:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2441673728 bytes 2024-09-18T01:34:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2485332992 bytes 2024-09-18T01:34:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2496105472 bytes 2024-09-18T01:34:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2535109632 bytes 2024-09-18T01:34:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2576929792 bytes 2024-09-18T01:34:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2631418880 bytes 2024-09-18T01:34:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2690091008 bytes 2024-09-18T01:34:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2741822464 bytes 2024-09-18T01:34:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2768055296 bytes 2024-09-18T01:34:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2827138048 bytes 2024-09-18T01:34:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2859186176 bytes 2024-09-18T01:34:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2940469248 bytes 2024-09-18T01:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 2957052928 bytes 2024-09-18T01:34:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3019660288 bytes 2024-09-18T01:34:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3116493824 bytes 2024-09-18T01:34:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3125508096 bytes 2024-09-18T01:34:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3246242816 bytes 2024-09-18T01:34:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3303354368 bytes 2024-09-18T01:34:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3407754240 bytes 2024-09-18T01:34:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3499379712 bytes 2024-09-18T01:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3549326336 bytes 2024-09-18T01:34:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3568371712 bytes 2024-09-18T01:34:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3642675200 bytes 2024-09-18T01:34:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3668463616 bytes 2024-09-18T01:34:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3679278080 bytes 2024-09-18T01:34:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3733446656 bytes 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_40 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-18T01:34:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:34:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_40': 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 145b81cd750d 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.145b81cd750d 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:34:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:34:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:33 2024 2024-09-18T01:34:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:33 2024 2024-09-18T01:34:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:33 2024 2024-09-18T01:34:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:34:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:34:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: d28cc690-e321-4677-9f77-fba5372ae4cd 2024-09-18T01:34:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:34:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:34:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-18T01:34:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:34:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:34:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:34:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:34:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:34:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 66190336 bytes 2024-09-18T01:34:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 129775616 bytes 2024-09-18T01:34:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 167028736 bytes 2024-09-18T01:34:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 208124928 bytes 2024-09-18T01:34:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 254829568 bytes 2024-09-18T01:34:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 303756288 bytes 2024-09-18T01:34:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 320048128 bytes 2024-09-18T01:34:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 359664640 bytes 2024-09-18T01:34:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 484343808 bytes 2024-09-18T01:34:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 520994816 bytes 2024-09-18T01:34:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 554925056 bytes 2024-09-18T01:34:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 629599232 bytes 2024-09-18T01:34:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 693932032 bytes 2024-09-18T01:34:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 750471168 bytes 2024-09-18T01:34:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 751092736 bytes 2024-09-18T01:34:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 763836416 bytes 2024-09-18T01:34:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 810023936 bytes 2024-09-18T01:34:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 869451776 bytes 2024-09-18T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 938971136 bytes 2024-09-18T01:34:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1035094016 bytes 2024-09-18T01:34:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1053425664 bytes 2024-09-18T01:34:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1101034496 bytes 2024-09-18T01:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1135739904 bytes 2024-09-18T01:34:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1197305856 bytes 2024-09-18T01:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1223066624 bytes 2024-09-18T01:34:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1248505856 bytes 2024-09-18T01:34:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1285824512 bytes 2024-09-18T01:34:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1314757632 bytes 2024-09-18T01:34:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1363982336 bytes 2024-09-18T01:34:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1439700992 bytes 2024-09-18T01:34:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1506569216 bytes 2024-09-18T01:34:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1567581184 bytes 2024-09-18T01:34:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1678824448 bytes 2024-09-18T01:34:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1701504000 bytes 2024-09-18T01:34:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1712176128 bytes 2024-09-18T01:34:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1784339456 bytes 2024-09-18T01:34:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1794499584 bytes 2024-09-18T01:34:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1827906560 bytes 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_41 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:34:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:34:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-18T01:34:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:34:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:34:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:34:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:34:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2024-09-18T01:34:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:34:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:34:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-18T01:34:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:34:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-18T01:34:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:34:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-18T01:34:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_41': 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 14673ac6618e 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.14673ac6618e 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:38 2024 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:38 2024 2024-09-18T01:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:38 2024 2024-09-18T01:34:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:34:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:34:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 5be9e68b-589f-432a-ad64-9c1d14313b4c 2024-09-18T01:34:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:34:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:34:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-18T01:34:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:34:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:34:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=13 2024-09-18T01:34:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:34:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 54820864 bytes 2024-09-18T01:34:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 131929088 bytes 2024-09-18T01:34:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 154363904 bytes 2024-09-18T01:34:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 171484160 bytes 2024-09-18T01:34:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 199007232 bytes 2024-09-18T01:34:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 285758464 bytes 2024-09-18T01:34:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 318481408 bytes 2024-09-18T01:34:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 405599232 bytes 2024-09-18T01:34:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 443831296 bytes 2024-09-18T01:34:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 464126976 bytes 2024-09-18T01:34:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 476379136 bytes 2024-09-18T01:34:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 524919808 bytes 2024-09-18T01:34:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 609958912 bytes 2024-09-18T01:34:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 690849792 bytes 2024-09-18T01:34:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 724160512 bytes 2024-09-18T01:34:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 748232704 bytes 2024-09-18T01:34:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 820066304 bytes 2024-09-18T01:34:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 856285184 bytes 2024-09-18T01:34:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 908315648 bytes 2024-09-18T01:34:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 922545152 bytes 2024-09-18T01:34:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 948677632 bytes 2024-09-18T01:34:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1049873408 bytes 2024-09-18T01:34:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1120268288 bytes 2024-09-18T01:34:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1128457216 bytes 2024-09-18T01:34:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1204431872 bytes 2024-09-18T01:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1233513472 bytes 2024-09-18T01:34:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1234128896 bytes 2024-09-18T01:34:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1252815872 bytes 2024-09-18T01:34:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1253406720 bytes 2024-09-18T01:34:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1254023168 bytes 2024-09-18T01:34:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1376376832 bytes 2024-09-18T01:34:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1411158016 bytes 2024-09-18T01:34:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1473142784 bytes 2024-09-18T01:34:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1525885952 bytes 2024-09-18T01:34:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1574777856 bytes 2024-09-18T01:34:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1642853376 bytes 2024-09-18T01:34:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1689257984 bytes 2024-09-18T01:34:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1725632512 bytes 2024-09-18T01:34:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1762680832 bytes 2024-09-18T01:34:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1786372096 bytes 2024-09-18T01:34:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1845275648 bytes 2024-09-18T01:34:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1850482688 bytes 2024-09-18T01:34:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1909001216 bytes 2024-09-18T01:34:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1932835840 bytes 2024-09-18T01:34:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1946003456 bytes 2024-09-18T01:34:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1973576704 bytes 2024-09-18T01:34:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2017275904 bytes 2024-09-18T01:34:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2074124288 bytes 2024-09-18T01:34:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2117026816 bytes 2024-09-18T01:34:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2142015488 bytes 2024-09-18T01:34:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2195223552 bytes 2024-09-18T01:34:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2211535872 bytes 2024-09-18T01:34:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2242019328 bytes 2024-09-18T01:34:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2316034048 bytes 2024-09-18T01:34:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2317560832 bytes 2024-09-18T01:34:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2350764032 bytes 2024-09-18T01:34:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2351369216 bytes 2024-09-18T01:34:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2383679488 bytes 2024-09-18T01:34:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2442191872 bytes 2024-09-18T01:34:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2511828992 bytes 2024-09-18T01:34:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2525477888 bytes 2024-09-18T01:34:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2526075904 bytes 2024-09-18T01:34:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2569012224 bytes 2024-09-18T01:34:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2616541184 bytes 2024-09-18T01:34:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2668480512 bytes 2024-09-18T01:34:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2670253056 bytes 2024-09-18T01:34:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2730985472 bytes 2024-09-18T01:34:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2758355968 bytes 2024-09-18T01:34:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2805567488 bytes 2024-09-18T01:34:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2823812096 bytes 2024-09-18T01:34:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2852227072 bytes 2024-09-18T01:34:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2857302016 bytes 2024-09-18T01:34:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2925526016 bytes 2024-09-18T01:34:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 2996725760 bytes 2024-09-18T01:34:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3032890368 bytes 2024-09-18T01:34:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3052099584 bytes 2024-09-18T01:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3108397056 bytes 2024-09-18T01:34:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3126650880 bytes 2024-09-18T01:34:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3230871552 bytes 2024-09-18T01:34:49.734 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3788241588038792) > weight (0.25) 2024-09-18T01:34:49.734 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5881657081727459) > weight (0.25) 2024-09-18T01:34:49.734 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7476863050033209) > weight (0.25) 2024-09-18T01:34:49.735 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7031160727331045) > weight (0.25) 2024-09-18T01:34:49.735 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-09-18T01:34:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3270851584 bytes 2024-09-18T01:34:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3275242496 bytes 2024-09-18T01:34:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3325755392 bytes 2024-09-18T01:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3395358720 bytes 2024-09-18T01:34:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3421263872 bytes 2024-09-18T01:34:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3462566912 bytes 2024-09-18T01:34:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3496448000 bytes 2024-09-18T01:34:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3570353152 bytes 2024-09-18T01:34:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3606882304 bytes 2024-09-18T01:34:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3670059008 bytes 2024-09-18T01:34:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3701512192 bytes 2024-09-18T01:34:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3762471936 bytes 2024-09-18T01:34:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3798139904 bytes 2024-09-18T01:34:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 3877257216 bytes 2024-09-18T01:34:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 3933443072 bytes 2024-09-18T01:34:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 3994428416 bytes 2024-09-18T01:34:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4028571648 bytes 2024-09-18T01:34:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4046742528 bytes 2024-09-18T01:34:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4075984896 bytes 2024-09-18T01:34:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4135947264 bytes 2024-09-18T01:34:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4196625408 bytes 2024-09-18T01:34:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_42 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:34:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-18T01:34:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2024-09-18T01:34:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:34:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:34:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-18T01:34:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:34:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-18T01:34:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:34:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_42': 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 147331da2721 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.147331da2721 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:34:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:34:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:52 2024 2024-09-18T01:34:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:52 2024 2024-09-18T01:34:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:52 2024 2024-09-18T01:34:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:34:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:34:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: f8e75016-c79a-43fb-b490-6cd4ba3ee983 2024-09-18T01:34:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:34:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:34:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-18T01:34:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:34:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:34:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:34:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:34:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:34:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 4254720 bytes 2024-09-18T01:34:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 43734016 bytes 2024-09-18T01:34:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 103924736 bytes 2024-09-18T01:34:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 149498880 bytes 2024-09-18T01:34:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 195233792 bytes 2024-09-18T01:34:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 267341824 bytes 2024-09-18T01:34:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 267963392 bytes 2024-09-18T01:34:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 365346816 bytes 2024-09-18T01:34:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 392786944 bytes 2024-09-18T01:34:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 410334208 bytes 2024-09-18T01:34:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 440344576 bytes 2024-09-18T01:34:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 465701888 bytes 2024-09-18T01:34:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 502991872 bytes 2024-09-18T01:34:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 552982528 bytes 2024-09-18T01:34:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 599555072 bytes 2024-09-18T01:34:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 640670720 bytes 2024-09-18T01:34:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 688057344 bytes 2024-09-18T01:34:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 738412544 bytes 2024-09-18T01:34:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 834481152 bytes 2024-09-18T01:34:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 851746816 bytes 2024-09-18T01:34:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 852382720 bytes 2024-09-18T01:34:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 852995072 bytes 2024-09-18T01:34:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 941816832 bytes 2024-09-18T01:34:56.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1002536960 bytes 2024-09-18T01:34:57.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1111326720 bytes 2024-09-18T01:34:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1158461440 bytes 2024-09-18T01:34:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1197712384 bytes 2024-09-18T01:34:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1284268032 bytes 2024-09-18T01:34:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1304211456 bytes 2024-09-18T01:34:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1323350016 bytes 2024-09-18T01:34:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1405443072 bytes 2024-09-18T01:34:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1479587840 bytes 2024-09-18T01:34:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1556694016 bytes 2024-09-18T01:34:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1594292224 bytes 2024-09-18T01:34:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1686611968 bytes 2024-09-18T01:34:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1687230464 bytes 2024-09-18T01:34:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1700282368 bytes 2024-09-18T01:34:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1739768832 bytes 2024-09-18T01:34:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1768365056 bytes 2024-09-18T01:34:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1795053568 bytes 2024-09-18T01:34:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1820700672 bytes 2024-09-18T01:34:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1850569728 bytes 2024-09-18T01:34:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1942247424 bytes 2024-09-18T01:34:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1983734784 bytes 2024-09-18T01:34:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2045713408 bytes 2024-09-18T01:34:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2068107264 bytes 2024-09-18T01:34:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2086080512 bytes 2024-09-18T01:34:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:34:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_43 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:34:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:34:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:34:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:34:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2024-09-18T01:34:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:34:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:34:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:34:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-18T01:34:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:34:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:34:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:34:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:34:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-18T01:35:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:35:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-18T01:35:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_43': 2024-09-18T01:35:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:35:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:35:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 147f2120659c 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.147f2120659c 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:59 2024 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:59 2024 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:59 2024 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 38e4e214-f1ca-4e80-ac69-0b111c7bccc1 2024-09-18T01:35:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:35:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:35:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-18T01:35:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:35:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:35:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=8 2024-09-18T01:35:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:35:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:35:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 11456512 bytes 2024-09-18T01:35:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 71874560 bytes 2024-09-18T01:35:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 132653056 bytes 2024-09-18T01:35:01.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 144949248 bytes 2024-09-18T01:35:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 145546240 bytes 2024-09-18T01:35:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 185779200 bytes 2024-09-18T01:35:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 228345856 bytes 2024-09-18T01:35:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 339753984 bytes 2024-09-18T01:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 348317696 bytes 2024-09-18T01:35:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 407092224 bytes 2024-09-18T01:35:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 480611328 bytes 2024-09-18T01:35:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 519825408 bytes 2024-09-18T01:35:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 566938624 bytes 2024-09-18T01:35:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 652338176 bytes 2024-09-18T01:35:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 698742784 bytes 2024-09-18T01:35:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 717904896 bytes 2024-09-18T01:35:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 743620608 bytes 2024-09-18T01:35:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 779070464 bytes 2024-09-18T01:35:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 838406144 bytes 2024-09-18T01:35:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 884208640 bytes 2024-09-18T01:35:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 968955904 bytes 2024-09-18T01:35:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 969593856 bytes 2024-09-18T01:35:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1008133120 bytes 2024-09-18T01:35:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1044775936 bytes 2024-09-18T01:35:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1115419648 bytes 2024-09-18T01:35:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1149602816 bytes 2024-09-18T01:35:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1202157568 bytes 2024-09-18T01:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1235146752 bytes 2024-09-18T01:35:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1269679104 bytes 2024-09-18T01:35:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1276201984 bytes 2024-09-18T01:35:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1308585984 bytes 2024-09-18T01:35:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1344562176 bytes 2024-09-18T01:35:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1370135552 bytes 2024-09-18T01:35:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1420475392 bytes 2024-09-18T01:35:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1460923392 bytes 2024-09-18T01:35:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1500879872 bytes 2024-09-18T01:35:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1545319424 bytes 2024-09-18T01:35:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1590023168 bytes 2024-09-18T01:35:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1654105088 bytes 2024-09-18T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1683600384 bytes 2024-09-18T01:35:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1743193088 bytes 2024-09-18T01:35:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1792499712 bytes 2024-09-18T01:35:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1840032768 bytes 2024-09-18T01:35:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1842609152 bytes 2024-09-18T01:35:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1851790336 bytes 2024-09-18T01:35:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1881460736 bytes 2024-09-18T01:35:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1911145472 bytes 2024-09-18T01:35:07.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1999974400 bytes 2024-09-18T01:35:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2066724864 bytes 2024-09-18T01:35:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2149464064 bytes 2024-09-18T01:35:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2167791616 bytes 2024-09-18T01:35:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2266909696 bytes 2024-09-18T01:35:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2324936704 bytes 2024-09-18T01:35:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2346562560 bytes 2024-09-18T01:35:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2379566080 bytes 2024-09-18T01:35:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2409400320 bytes 2024-09-18T01:35:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2426509312 bytes 2024-09-18T01:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2454364160 bytes 2024-09-18T01:35:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2469894144 bytes 2024-09-18T01:35:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2526451712 bytes 2024-09-18T01:35:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2569900032 bytes 2024-09-18T01:35:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2657598464 bytes 2024-09-18T01:35:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:35:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:35:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:35:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:35:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_44 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:35:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:35:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:35:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:35:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-18T01:35:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:35:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-18T01:35:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_44': 2024-09-18T01:35:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:35:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 148b1fa84543 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.148b1fa84543 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:08 2024 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:08 2024 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:08 2024 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:35:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: f9dd9870-23fe-40ab-a93a-2441b3c7b9d1 2024-09-18T01:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-18T01:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:35:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:35:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:35:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:35:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 26472448 bytes 2024-09-18T01:35:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 49460224 bytes 2024-09-18T01:35:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 107921408 bytes 2024-09-18T01:35:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 186669056 bytes 2024-09-18T01:35:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 229499904 bytes 2024-09-18T01:35:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 268286976 bytes 2024-09-18T01:35:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 281089024 bytes 2024-09-18T01:35:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 307698688 bytes 2024-09-18T01:35:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 314422272 bytes 2024-09-18T01:35:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 382836736 bytes 2024-09-18T01:35:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 405149696 bytes 2024-09-18T01:35:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 492256256 bytes 2024-09-18T01:35:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 533828608 bytes 2024-09-18T01:35:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 649158656 bytes 2024-09-18T01:35:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 689125376 bytes 2024-09-18T01:35:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 755777536 bytes 2024-09-18T01:35:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 756421632 bytes 2024-09-18T01:35:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 757021696 bytes 2024-09-18T01:35:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 775918592 bytes 2024-09-18T01:35:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 841012224 bytes 2024-09-18T01:35:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 861308928 bytes 2024-09-18T01:35:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 883955712 bytes 2024-09-18T01:35:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 910325760 bytes 2024-09-18T01:35:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 966887424 bytes 2024-09-18T01:35:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1094546432 bytes 2024-09-18T01:35:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1142714368 bytes 2024-09-18T01:35:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1165982720 bytes 2024-09-18T01:35:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1221950464 bytes 2024-09-18T01:35:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1316936704 bytes 2024-09-18T01:35:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1401237504 bytes 2024-09-18T01:35:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1456036864 bytes 2024-09-18T01:35:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1541848064 bytes 2024-09-18T01:35:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1574334464 bytes 2024-09-18T01:35:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1628688384 bytes 2024-09-18T01:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1641913344 bytes 2024-09-18T01:35:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1642508288 bytes 2024-09-18T01:35:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1662685184 bytes 2024-09-18T01:35:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1723361280 bytes 2024-09-18T01:35:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1758301184 bytes 2024-09-18T01:35:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:35:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_45 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2024-09-18T01:35:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:35:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:35:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-18T01:35:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:35:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-18T01:35:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:35:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-18T01:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_45': 2024-09-18T01:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 1497b783de31 2024-09-18T01:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.1497b783de31 2024-09-18T01:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:14 2024 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:14 2024 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:14 2024 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 454fe261-0829-4a00-bce9-9223fa6aeed6 2024-09-18T01:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:35:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:35:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-18T01:35:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:35:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:35:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 42970112 bytes 2024-09-18T01:35:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 77177856 bytes 2024-09-18T01:35:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 124414976 bytes 2024-09-18T01:35:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 127427584 bytes 2024-09-18T01:35:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 173097984 bytes 2024-09-18T01:35:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 214334464 bytes 2024-09-18T01:35:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 258688000 bytes 2024-09-18T01:35:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 302469120 bytes 2024-09-18T01:35:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 364648448 bytes 2024-09-18T01:35:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 438999040 bytes 2024-09-18T01:35:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 482344960 bytes 2024-09-18T01:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 570846208 bytes 2024-09-18T01:35:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 574117888 bytes 2024-09-18T01:35:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 590860288 bytes 2024-09-18T01:35:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 603632640 bytes 2024-09-18T01:35:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 641562624 bytes 2024-09-18T01:35:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 656166912 bytes 2024-09-18T01:35:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 691384320 bytes 2024-09-18T01:35:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 732826624 bytes 2024-09-18T01:35:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 748740608 bytes 2024-09-18T01:35:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 832115712 bytes 2024-09-18T01:35:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 864040960 bytes 2024-09-18T01:35:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 907830272 bytes 2024-09-18T01:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 930416640 bytes 2024-09-18T01:35:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 988044288 bytes 2024-09-18T01:35:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1006748672 bytes 2024-09-18T01:35:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1091969024 bytes 2024-09-18T01:35:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1159031808 bytes 2024-09-18T01:35:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1188077568 bytes 2024-09-18T01:35:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1231300608 bytes 2024-09-18T01:35:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1290184704 bytes 2024-09-18T01:35:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1345572864 bytes 2024-09-18T01:35:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1434255360 bytes 2024-09-18T01:35:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1500153856 bytes 2024-09-18T01:35:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1500773376 bytes 2024-09-18T01:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1501383680 bytes 2024-09-18T01:35:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1556914176 bytes 2024-09-18T01:35:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1599935488 bytes 2024-09-18T01:35:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1650816000 bytes 2024-09-18T01:35:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1669439488 bytes 2024-09-18T01:35:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1741981696 bytes 2024-09-18T01:35:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1807502336 bytes 2024-09-18T01:35:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1859831808 bytes 2024-09-18T01:35:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1898685440 bytes 2024-09-18T01:35:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1905121280 bytes 2024-09-18T01:35:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1939490816 bytes 2024-09-18T01:35:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2024705024 bytes 2024-09-18T01:35:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2088836096 bytes 2024-09-18T01:35:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2115806208 bytes 2024-09-18T01:35:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2119726080 bytes 2024-09-18T01:35:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2141617152 bytes 2024-09-18T01:35:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2204176384 bytes 2024-09-18T01:35:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2264541184 bytes 2024-09-18T01:35:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2290105344 bytes 2024-09-18T01:35:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2337354752 bytes 2024-09-18T01:35:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2398678016 bytes 2024-09-18T01:35:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2415211520 bytes 2024-09-18T01:35:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2495547392 bytes 2024-09-18T01:35:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2549131264 bytes 2024-09-18T01:35:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2558391296 bytes 2024-09-18T01:35:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2587657216 bytes 2024-09-18T01:35:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2744387584 bytes 2024-09-18T01:35:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2767644672 bytes 2024-09-18T01:35:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2858669056 bytes 2024-09-18T01:35:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2864464896 bytes 2024-09-18T01:35:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2904759296 bytes 2024-09-18T01:35:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2982975488 bytes 2024-09-18T01:35:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3041762304 bytes 2024-09-18T01:35:24.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3092609024 bytes 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_46 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:35:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:35:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:35:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:35:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2024-09-18T01:35:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:35:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:35:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-18T01:35:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:35:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-18T01:35:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:35:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-18T01:35:25.939 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6844319580024066) > weight (0.25) 2024-09-18T01:35:25.939 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2977433640314745) > weight (0.25) 2024-09-18T01:35:25.939 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4293144473716488) > weight (0.25) 2024-09-18T01:35:25.939 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5354350627078767) > weight (0.25) 2024-09-18T01:35:25.939 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-18T01:35:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_46': 2024-09-18T01:35:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:35:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:35:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:35:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 14a33a105899 2024-09-18T01:35:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.14a33a105899 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:24 2024 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:24 2024 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:24 2024 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 891bfe82-6886-4816-b990-e7e59c1ed8c5 2024-09-18T01:35:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:35:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:35:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-18T01:35:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:35:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:35:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=11 2024-09-18T01:35:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:35:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:35:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 609280 bytes 2024-09-18T01:35:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 1793024 bytes 2024-09-18T01:35:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 47763456 bytes 2024-09-18T01:35:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 91364352 bytes 2024-09-18T01:35:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 100768768 bytes 2024-09-18T01:35:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 110573568 bytes 2024-09-18T01:35:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 200159232 bytes 2024-09-18T01:35:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 233697280 bytes 2024-09-18T01:35:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 322624512 bytes 2024-09-18T01:35:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 371350528 bytes 2024-09-18T01:35:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 414731264 bytes 2024-09-18T01:35:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 469923840 bytes 2024-09-18T01:35:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 500689920 bytes 2024-09-18T01:35:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 538862592 bytes 2024-09-18T01:35:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 578390016 bytes 2024-09-18T01:35:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 605440000 bytes 2024-09-18T01:35:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 685151232 bytes 2024-09-18T01:35:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 704672768 bytes 2024-09-18T01:35:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 779164672 bytes 2024-09-18T01:35:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 805074944 bytes 2024-09-18T01:35:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 829625344 bytes 2024-09-18T01:35:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 844929024 bytes 2024-09-18T01:35:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 875857920 bytes 2024-09-18T01:35:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 928885760 bytes 2024-09-18T01:35:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 966807552 bytes 2024-09-18T01:35:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 993011712 bytes 2024-09-18T01:35:29.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1029094400 bytes 2024-09-18T01:35:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1120463872 bytes 2024-09-18T01:35:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1138242560 bytes 2024-09-18T01:35:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1154658304 bytes 2024-09-18T01:35:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1227635712 bytes 2024-09-18T01:35:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1324179456 bytes 2024-09-18T01:35:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1344020480 bytes 2024-09-18T01:35:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1384485888 bytes 2024-09-18T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1488692224 bytes 2024-09-18T01:35:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1489318912 bytes 2024-09-18T01:35:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1549480960 bytes 2024-09-18T01:35:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1593890816 bytes 2024-09-18T01:35:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1620377600 bytes 2024-09-18T01:35:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1652452352 bytes 2024-09-18T01:35:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1679182848 bytes 2024-09-18T01:35:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1709562880 bytes 2024-09-18T01:35:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1722216448 bytes 2024-09-18T01:35:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1727151104 bytes 2024-09-18T01:35:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1772477440 bytes 2024-09-18T01:35:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1804215296 bytes 2024-09-18T01:35:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1887178752 bytes 2024-09-18T01:35:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1920854016 bytes 2024-09-18T01:35:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1990634496 bytes 2024-09-18T01:35:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2055996416 bytes 2024-09-18T01:35:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2103930880 bytes 2024-09-18T01:35:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2125689856 bytes 2024-09-18T01:35:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2146928640 bytes 2024-09-18T01:35:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2185835520 bytes 2024-09-18T01:35:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2239595520 bytes 2024-09-18T01:35:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2260776960 bytes 2024-09-18T01:35:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2286147584 bytes 2024-09-18T01:35:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2335094784 bytes 2024-09-18T01:35:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2355402752 bytes 2024-09-18T01:35:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2418514944 bytes 2024-09-18T01:35:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2481722368 bytes 2024-09-18T01:35:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2527835136 bytes 2024-09-18T01:35:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2556014592 bytes 2024-09-18T01:35:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2619058176 bytes 2024-09-18T01:35:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2732128256 bytes 2024-09-18T01:35:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2772299776 bytes 2024-09-18T01:35:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2783262720 bytes 2024-09-18T01:35:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2809274368 bytes 2024-09-18T01:35:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2847883264 bytes 2024-09-18T01:35:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2907526144 bytes 2024-09-18T01:35:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2941614080 bytes 2024-09-18T01:35:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3001050112 bytes 2024-09-18T01:35:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3054197760 bytes 2024-09-18T01:35:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3132654592 bytes 2024-09-18T01:35:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3137889280 bytes 2024-09-18T01:35:35.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3213775872 bytes 2024-09-18T01:35:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3274804224 bytes 2024-09-18T01:35:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3279967232 bytes 2024-09-18T01:35:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3290590208 bytes 2024-09-18T01:35:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3317011456 bytes 2024-09-18T01:35:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3346923520 bytes 2024-09-18T01:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3431185408 bytes 2024-09-18T01:35:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3487602688 bytes 2024-09-18T01:35:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3529643008 bytes 2024-09-18T01:35:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3557910528 bytes 2024-09-18T01:35:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3601543168 bytes 2024-09-18T01:35:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3678541824 bytes 2024-09-18T01:35:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3683641344 bytes 2024-09-18T01:35:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3737985024 bytes 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_47 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:35:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:35:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2024-09-18T01:35:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:35:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:35:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-18T01:35:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:35:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-18T01:35:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:35:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-18T01:35:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_47': 2024-09-18T01:35:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:35:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:35:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:35:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 14afc78c5647 2024-09-18T01:35:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.14afc78c5647 2024-09-18T01:35:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:35:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:36 2024 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:36 2024 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:36 2024 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: c36486c3-b203-4110-86f0-cb987a216048 2024-09-18T01:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:35:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:35:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-18T01:35:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:35:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:35:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=12 2024-09-18T01:35:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:35:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:35:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 60184576 bytes 2024-09-18T01:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 105237504 bytes 2024-09-18T01:35:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 156174336 bytes 2024-09-18T01:35:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 206697472 bytes 2024-09-18T01:35:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 286244864 bytes 2024-09-18T01:35:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 309172224 bytes 2024-09-18T01:35:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 351559680 bytes 2024-09-18T01:35:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 404379648 bytes 2024-09-18T01:35:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 490773504 bytes 2024-09-18T01:35:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 494648320 bytes 2024-09-18T01:35:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 544991232 bytes 2024-09-18T01:35:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 576286720 bytes 2024-09-18T01:35:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 640512000 bytes 2024-09-18T01:35:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 692890624 bytes 2024-09-18T01:35:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 741991424 bytes 2024-09-18T01:35:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 812424192 bytes 2024-09-18T01:35:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 893988864 bytes 2024-09-18T01:35:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 914963456 bytes 2024-09-18T01:35:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 988701696 bytes 2024-09-18T01:35:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 1019219968 bytes 2024-09-18T01:35:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1025292288 bytes 2024-09-18T01:35:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1085232128 bytes 2024-09-18T01:35:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1101168640 bytes 2024-09-18T01:35:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1102818304 bytes 2024-09-18T01:35:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1135488000 bytes 2024-09-18T01:35:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1179846656 bytes 2024-09-18T01:35:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1219222528 bytes 2024-09-18T01:35:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1264415744 bytes 2024-09-18T01:35:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1347490816 bytes 2024-09-18T01:35:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1368839168 bytes 2024-09-18T01:35:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1414347776 bytes 2024-09-18T01:35:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1434897408 bytes 2024-09-18T01:35:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1469680640 bytes 2024-09-18T01:35:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1507825664 bytes 2024-09-18T01:35:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1547470848 bytes 2024-09-18T01:35:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1597587456 bytes 2024-09-18T01:35:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1610351616 bytes 2024-09-18T01:35:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1659325440 bytes 2024-09-18T01:35:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1688307712 bytes 2024-09-18T01:35:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1720701952 bytes 2024-09-18T01:35:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1748581376 bytes 2024-09-18T01:35:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1782017024 bytes 2024-09-18T01:35:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1845410816 bytes 2024-09-18T01:35:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1934139392 bytes 2024-09-18T01:35:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1972154368 bytes 2024-09-18T01:35:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2017119232 bytes 2024-09-18T01:35:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2057682944 bytes 2024-09-18T01:35:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2095241216 bytes 2024-09-18T01:35:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2142690304 bytes 2024-09-18T01:35:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2182651904 bytes 2024-09-18T01:35:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2230182912 bytes 2024-09-18T01:35:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2288841728 bytes 2024-09-18T01:35:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2295155712 bytes 2024-09-18T01:35:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2332724224 bytes 2024-09-18T01:35:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2403396608 bytes 2024-09-18T01:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2444264448 bytes 2024-09-18T01:35:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2499446784 bytes 2024-09-18T01:35:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2540573696 bytes 2024-09-18T01:35:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2578528256 bytes 2024-09-18T01:35:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2625204224 bytes 2024-09-18T01:35:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2718607360 bytes 2024-09-18T01:35:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2761223168 bytes 2024-09-18T01:35:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2793288704 bytes 2024-09-18T01:35:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2860313600 bytes 2024-09-18T01:35:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2945581056 bytes 2024-09-18T01:35:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2982732800 bytes 2024-09-18T01:35:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2985622528 bytes 2024-09-18T01:35:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2986261504 bytes 2024-09-18T01:35:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2996688896 bytes 2024-09-18T01:35:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3003614208 bytes 2024-09-18T01:35:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3032627200 bytes 2024-09-18T01:35:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3044802560 bytes 2024-09-18T01:35:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3054189568 bytes 2024-09-18T01:35:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3109558272 bytes 2024-09-18T01:35:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3194587136 bytes 2024-09-18T01:35:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3260395520 bytes 2024-09-18T01:35:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3282842624 bytes 2024-09-18T01:35:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3294447616 bytes 2024-09-18T01:35:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3329929216 bytes 2024-09-18T01:35:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3355185152 bytes 2024-09-18T01:35:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3442065408 bytes 2024-09-18T01:35:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3492802560 bytes 2024-09-18T01:35:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3563187200 bytes 2024-09-18T01:35:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3599223808 bytes 2024-09-18T01:35:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3611509760 bytes 2024-09-18T01:35:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3631229952 bytes 2024-09-18T01:35:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3653622784 bytes 2024-09-18T01:35:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3703542784 bytes 2024-09-18T01:35:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3730540544 bytes 2024-09-18T01:35:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3828582400 bytes 2024-09-18T01:35:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3941117952 bytes 2024-09-18T01:35:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4030025728 bytes 2024-09-18T01:35:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4070949888 bytes 2024-09-18T01:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_48 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2024-09-18T01:35:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-18T01:35:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:35:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_48': 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 14bbfe2422a 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.14bbfe2422a 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:50 2024 2024-09-18T01:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:50 2024 2024-09-18T01:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:50 2024 2024-09-18T01:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 9bca644c-78a0-405d-b934-3bd6906b0d98 2024-09-18T01:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:35:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:35:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-18T01:35:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:35:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:35:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:35:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:35:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:35:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 57351168 bytes 2024-09-18T01:35:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 107457536 bytes 2024-09-18T01:35:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 136377344 bytes 2024-09-18T01:35:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 180702208 bytes 2024-09-18T01:35:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 201810944 bytes 2024-09-18T01:35:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 230411264 bytes 2024-09-18T01:35:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 300810240 bytes 2024-09-18T01:35:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 352798720 bytes 2024-09-18T01:35:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 384345088 bytes 2024-09-18T01:35:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 490027008 bytes 2024-09-18T01:35:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 543616000 bytes 2024-09-18T01:35:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 554467328 bytes 2024-09-18T01:35:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 625144832 bytes 2024-09-18T01:35:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 658122752 bytes 2024-09-18T01:35:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 662912000 bytes 2024-09-18T01:35:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 730246144 bytes 2024-09-18T01:35:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 796489728 bytes 2024-09-18T01:35:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 880885760 bytes 2024-09-18T01:35:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 898070528 bytes 2024-09-18T01:35:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 983325696 bytes 2024-09-18T01:35:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1006792704 bytes 2024-09-18T01:35:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1051211776 bytes 2024-09-18T01:35:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1107990528 bytes 2024-09-18T01:35:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1137637376 bytes 2024-09-18T01:35:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1219447808 bytes 2024-09-18T01:35:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1269043200 bytes 2024-09-18T01:35:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1320938496 bytes 2024-09-18T01:35:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1356935168 bytes 2024-09-18T01:35:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1414513664 bytes 2024-09-18T01:35:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1437112320 bytes 2024-09-18T01:35:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1503288320 bytes 2024-09-18T01:35:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1508723712 bytes 2024-09-18T01:35:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1553471488 bytes 2024-09-18T01:35:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1602158592 bytes 2024-09-18T01:35:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1697529856 bytes 2024-09-18T01:35:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1726089216 bytes 2024-09-18T01:35:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1726682112 bytes 2024-09-18T01:35:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:35:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:35:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:35:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:35:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_49 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-18T01:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:35:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:35:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:35:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:35:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:35:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:35:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:35:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-18T01:35:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:35:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-18T01:35:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_49': 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 14c79c7a03b2 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.14c79c7a03b2 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:56 2024 2024-09-18T01:35:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:56 2024 2024-09-18T01:35:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:56 2024 2024-09-18T01:35:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:35:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:35:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 2a71d26d-40fd-4228-8efb-d1e4955324d6 2024-09-18T01:35:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:35:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:35:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-18T01:35:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:35:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:35:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:35:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:35:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=10 2024-09-18T01:35:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:35:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:35:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 45820928 bytes 2024-09-18T01:35:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 72513536 bytes 2024-09-18T01:35:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 169166848 bytes 2024-09-18T01:35:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 218455040 bytes 2024-09-18T01:35:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 293708800 bytes 2024-09-18T01:35:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 325896192 bytes 2024-09-18T01:35:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 376012800 bytes 2024-09-18T01:35:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 400463872 bytes 2024-09-18T01:35:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 498462720 bytes 2024-09-18T01:35:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 530420736 bytes 2024-09-18T01:35:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 578048000 bytes 2024-09-18T01:35:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 604601344 bytes 2024-09-18T01:35:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 668592128 bytes 2024-09-18T01:35:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 701072384 bytes 2024-09-18T01:35:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 757884928 bytes 2024-09-18T01:35:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 815406080 bytes 2024-09-18T01:35:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 872046592 bytes 2024-09-18T01:35:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 883939328 bytes 2024-09-18T01:35:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 914300928 bytes 2024-09-18T01:35:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 959501312 bytes 2024-09-18T01:35:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 984119296 bytes 2024-09-18T01:35:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1009238016 bytes 2024-09-18T01:35:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1060085760 bytes 2024-09-18T01:36:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1080730624 bytes 2024-09-18T01:36:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1118806016 bytes 2024-09-18T01:36:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1182898176 bytes 2024-09-18T01:36:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1274049536 bytes 2024-09-18T01:36:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1322740736 bytes 2024-09-18T01:36:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1376243712 bytes 2024-09-18T01:36:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1458691072 bytes 2024-09-18T01:36:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1467390976 bytes 2024-09-18T01:36:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1500392448 bytes 2024-09-18T01:36:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1536970752 bytes 2024-09-18T01:36:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1608993792 bytes 2024-09-18T01:36:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1620538368 bytes 2024-09-18T01:36:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1665840128 bytes 2024-09-18T01:36:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1764152320 bytes 2024-09-18T01:36:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1860369408 bytes 2024-09-18T01:36:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1905276928 bytes 2024-09-18T01:36:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1949539328 bytes 2024-09-18T01:36:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1955409920 bytes 2024-09-18T01:36:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1956025344 bytes 2024-09-18T01:36:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2024208384 bytes 2024-09-18T01:36:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2106756096 bytes 2024-09-18T01:36:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2169118720 bytes 2024-09-18T01:36:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2180419584 bytes 2024-09-18T01:36:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2227280896 bytes 2024-09-18T01:36:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2275173376 bytes 2024-09-18T01:36:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2275793920 bytes 2024-09-18T01:36:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2340514816 bytes 2024-09-18T01:36:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2365800448 bytes 2024-09-18T01:36:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2371565568 bytes 2024-09-18T01:36:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2427254784 bytes 2024-09-18T01:36:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2443232256 bytes 2024-09-18T01:36:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2483264512 bytes 2024-09-18T01:36:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2577292288 bytes 2024-09-18T01:36:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2599294976 bytes 2024-09-18T01:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2677580800 bytes 2024-09-18T01:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2682283008 bytes 2024-09-18T01:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2711499776 bytes 2024-09-18T01:36:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2751358976 bytes 2024-09-18T01:36:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2809701376 bytes 2024-09-18T01:36:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2826198016 bytes 2024-09-18T01:36:05.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2879762432 bytes 2024-09-18T01:36:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2959192064 bytes 2024-09-18T01:36:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2959798272 bytes 2024-09-18T01:36:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2989176832 bytes 2024-09-18T01:36:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3010205696 bytes 2024-09-18T01:36:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3079440384 bytes 2024-09-18T01:36:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3102936064 bytes 2024-09-18T01:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3150531584 bytes 2024-09-18T01:36:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3165727744 bytes 2024-09-18T01:36:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3187155968 bytes 2024-09-18T01:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3221791744 bytes 2024-09-18T01:36:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3280397312 bytes 2024-09-18T01:36:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3315102720 bytes 2024-09-18T01:36:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3368731648 bytes 2024-09-18T01:36:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3443528704 bytes 2024-09-18T01:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_50 2024-09-18T01:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n snapshot ']' 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 128M ']' 2024-09-18T01:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=128M 2024-09-18T01:36:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:36:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2024-09-18T01:36:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T01:36:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' image = image ']' 2024-09-18T01:36:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-18T01:36:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:36:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:36:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-18T01:36:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:36:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_50': 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 14d3ce37c6d5 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.14d3ce37c6d5 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:36:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:36:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:36:07 2024 2024-09-18T01:36:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:36:07 2024 2024-09-18T01:36:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:36:07 2024 2024-09-18T01:36:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:36:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:36:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 03d12fa6-b2eb-4ce4-bb75-8944661eb0eb 2024-09-18T01:36:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:36:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T01:36:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-18T01:36:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:36:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:36:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=7 2024-09-18T01:36:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:36:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:36:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 72640512 bytes 2024-09-18T01:36:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 90029056 bytes 2024-09-18T01:36:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 125929472 bytes 2024-09-18T01:36:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 186114048 bytes 2024-09-18T01:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 195347456 bytes 2024-09-18T01:36:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 277577728 bytes 2024-09-18T01:36:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 328309760 bytes 2024-09-18T01:36:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 365514752 bytes 2024-09-18T01:36:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 379831296 bytes 2024-09-18T01:36:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 413855744 bytes 2024-09-18T01:36:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 432517120 bytes 2024-09-18T01:36:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 485049344 bytes 2024-09-18T01:36:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 518858752 bytes 2024-09-18T01:36:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 545727488 bytes 2024-09-18T01:36:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 564044800 bytes 2024-09-18T01:36:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 601789440 bytes 2024-09-18T01:36:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 638959616 bytes 2024-09-18T01:36:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 639566848 bytes 2024-09-18T01:36:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 686913536 bytes 2024-09-18T01:36:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 712400896 bytes 2024-09-18T01:36:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 713029632 bytes 2024-09-18T01:36:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 765414400 bytes 2024-09-18T01:36:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 791261184 bytes 2024-09-18T01:36:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 846198784 bytes 2024-09-18T01:36:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 850359296 bytes 2024-09-18T01:36:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 894316544 bytes 2024-09-18T01:36:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 924781568 bytes 2024-09-18T01:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 945202176 bytes 2024-09-18T01:36:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 965254144 bytes 2024-09-18T01:36:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 987534336 bytes 2024-09-18T01:36:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1020761088 bytes 2024-09-18T01:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1034976256 bytes 2024-09-18T01:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1045723136 bytes 2024-09-18T01:36:11.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1088105472 bytes 2024-09-18T01:36:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1119372288 bytes 2024-09-18T01:36:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1177296896 bytes 2024-09-18T01:36:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1219022848 bytes 2024-09-18T01:36:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1260605440 bytes 2024-09-18T01:36:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1396413440 bytes 2024-09-18T01:36:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1448726528 bytes 2024-09-18T01:36:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1494759424 bytes 2024-09-18T01:36:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1528765440 bytes 2024-09-18T01:36:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1538690048 bytes 2024-09-18T01:36:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1539289088 bytes 2024-09-18T01:36:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1587320832 bytes 2024-09-18T01:36:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1629104128 bytes 2024-09-18T01:36:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1645194240 bytes 2024-09-18T01:36:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1692748800 bytes 2024-09-18T01:36:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1769323520 bytes 2024-09-18T01:36:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1808817152 bytes 2024-09-18T01:36:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 1848690688 bytes 2024-09-18T01:36:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 1852059648 bytes 2024-09-18T01:36:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 1870493696 bytes 2024-09-18T01:36:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 1915311104 bytes 2024-09-18T01:36:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 1976656896 bytes 2024-09-18T01:36:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 1977262080 bytes 2024-09-18T01:36:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2004369408 bytes 2024-09-18T01:36:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2030283776 bytes 2024-09-18T01:36:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2052953088 bytes 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_count=50 2024-09-18T01:36:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:36:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local count 2024-09-18T01:36:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_count=0 2024-09-18T01:36:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:36:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 40 2024-09-18T01:36:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in `seq 1 40` 2024-09-18T01:36:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 1 -ne 1 2024-09-18T01:36:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-18T01:36:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep 'images: ' 2024-09-18T01:36:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cut '-d ' -f 2 2024-09-18T01:36:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: failed to query services: (13) Permission denied 2024-09-18T01:36:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ count=50 2024-09-18T01:36:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 50 = 50 2024-09-18T01:36:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-18T01:36:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:36:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state 2024-09-18T01:36:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 40 2024-09-18T01:36:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in `seq 1 40` 2024-09-18T01:36:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 1 -ne 1 2024-09-18T01:36:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-18T01:36:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep 'image health:' 2024-09-18T01:36:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cut '-d ' -f 3 2024-09-18T01:36:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: failed to query services: (13) Permission denied 2024-09-18T01:36:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=OK 2024-09-18T01:36:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test OK = ERROR 2024-09-18T01:36:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test OK = OK 2024-09-18T01:36:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-18T01:36:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_count=50 2024-09-18T01:36:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:36:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local count 2024-09-18T01:36:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_count=0 2024-09-18T01:36:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:36:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 40 2024-09-18T01:36:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in `seq 1 40` 2024-09-18T01:36:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 1 -ne 1 2024-09-18T01:36:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-18T01:36:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep 'images: ' 2024-09-18T01:36:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cut '-d ' -f 2 2024-09-18T01:36:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: failed to query services: (13) Permission denied 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ count=50 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 50 = 50 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state 2024-09-18T01:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 40 2024-09-18T01:36:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in `seq 1 40` 2024-09-18T01:36:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 1 -ne 1 2024-09-18T01:36:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-18T01:36:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep 'image health:' 2024-09-18T01:36:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cut '-d ' -f 3 2024-09-18T01:36:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: failed to query services: (13) Permission denied 2024-09-18T01:36:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=WARNING 2024-09-18T01:36:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test WARNING = ERROR 2024-09-18T01:36:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test WARNING = OK 2024-09-18T01:36:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in `seq 1 40` 2024-09-18T01:36:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 2 -ne 1 2024-09-18T01:36:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 30 2024-09-18T01:36:39.736 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-18T01:36:39.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-18T01:36:39.736 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-18T01:36:39.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:36:39.736+0000 7f9726c12640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 48008) UID: 1000 2024-09-18T01:36:42.737 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-18T01:36:42.737 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:36:42.738 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-18T01:36:42.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-18T01:36:42.738 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-18T01:36:42.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-18T01:36:42.741 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-09-18T01:36:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-18T01:36:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep 'image health:' 2024-09-18T01:36:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cut '-d ' -f 3 2024-09-18T01:36:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: failed to query services: (13) Permission denied 2024-09-18T01:36:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=OK 2024-09-18T01:36:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test OK = ERROR 2024-09-18T01:36:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test OK = OK 2024-09-18T01:36:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ testlog 'TEST: compare many images' 2024-09-18T01:36:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ tee -a /tmp/tmp.YxZgdY10x3/rbd-mirror.test.log 2024-09-18T01:36:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ date '+%F %T' 2024-09-18T01:36:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo 2024-09-18 01:36:45 TEST: compare many images 2024-09-18T01:36:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:2024-09-18 01:36:45 TEST: compare many images 2024-09-18T01:36:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 50 2024-09-18T01:36:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:36:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_1 2024-09-18T01:36:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-18T01:36:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:36:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:36:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:36:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-18T01:36:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:36:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:36:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:36:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:36:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:36:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-18T01:36:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:36:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:36:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:36:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:36:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:36:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:36:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:36:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:36:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:36:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:36:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:36:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-18T01:36:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-18T01:36:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:36:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:36:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:36:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-18T01:36:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:36:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:36:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:36:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:36:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:36:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:36:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:36:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:36:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:36:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-18T01:36:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-18T01:36:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:36:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:36:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:36:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-09-18T01:36:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:36:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:36:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:36:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:36:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:36:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:36:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:36:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:36:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:36:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-09-18T01:36:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-18T01:36:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:36:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_1 2024-09-18T01:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:36:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:36:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:36:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:36:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_1 2024-09-18T01:36:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:36:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:36:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:36:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:36:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-18T01:36:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:36:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-18T01:36:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_1.status 2024-09-18T01:36:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_1.status 2024-09-18T01:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-18T01:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_1.status 2024-09-18T01:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-18T01:36:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-18T01:36:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 86 2024-09-18T01:36:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_1.status 2024-09-18T01:36:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_1.status 2024-09-18T01:36:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-18T01:36:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:36:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=86 2024-09-18T01:36:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:36:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:36:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:36:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-18T01:36:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:36:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=35 2024-09-18T01:36:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 86 = 35 2024-09-18T01:36:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:36:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:36:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-18T01:36:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:36:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:36:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:36:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:36:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:36:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:36:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 86 = '' 2024-09-18T01:36:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:36:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:36:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-18T01:36:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:36:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:36:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:36:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:36:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:36:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:36:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 86 = '' 2024-09-18T01:36:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:36:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:36:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.status 2024-09-18T01:36:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-18T01:36:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:36:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=86 2024-09-18T01:36:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 86 = 86 2024-09-18T01:36:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-18T01:36:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:36:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:36:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:36:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:36:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-18T01:36:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_1': 2024-09-18T01:36:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:36:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:36:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:36:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5dceaba28 2024-09-18T01:36:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5dceaba28 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:28:31 2024 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:28:31 2024 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:28:31 2024 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: e824ac62-3a6a-4a7a-9f9f-f4f31117d463 2024-09-18T01:36:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:36:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-18T01:36:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:36:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:36:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:36:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_1.export 2024-09-18T01:36:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.export 2024-09-18T01:36:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_1.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.export 2024-09-18T01:36:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_1.export 2024-09-18T01:36:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:36:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.export 2024-09-18T01:36:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_1.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.export 2024-09-18T01:36:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_1.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_1.export 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_2 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:36:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:36:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:36:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-18T01:36:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:36:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:36:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:36:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-18T01:36:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:36:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:36:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:36:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:36:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:36:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:36:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:36:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:36:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:36:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:36:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-18T01:36:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:36:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:36:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:36:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:36:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:36:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:36:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:36:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-18T01:36:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-18T01:36:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:36:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:36:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:36:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:36:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:36:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:36:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:36:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:36:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-18T01:36:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-18T01:36:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:36:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:36:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:36:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-18T01:36:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:36:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:36:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:36:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:36:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:36:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:36:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:36:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:36:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:36:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:36:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-18T01:36:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-18T01:36:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:36:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:36:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:36:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-09-18T01:36:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:36:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:36:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:36:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:36:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:36:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:36:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:36:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-09-18T01:36:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:36:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:36:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:36:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:36:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:36:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:36:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:36:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:36:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-18T01:36:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_2.status 2024-09-18T01:36:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:36:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_2.status 2024-09-18T01:36:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-18T01:36:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_2.status 2024-09-18T01:36:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:36:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:36:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-18T01:36:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-18T01:36:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 88 2024-09-18T01:36:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_2.status 2024-09-18T01:36:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:36:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_2.status 2024-09-18T01:36:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-18T01:36:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:36:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=88 2024-09-18T01:36:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:36:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:36:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:36:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:36:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:36:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-18T01:36:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:36:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=36 2024-09-18T01:36:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 88 = 36 2024-09-18T01:36:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:36:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:36:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:36:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:36:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-18T01:36:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:36:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:36:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:36:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:36:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:36:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:36:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 88 = '' 2024-09-18T01:36:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:36:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:36:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:36:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:36:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:36:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:36:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:36:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-18T01:36:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:36:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:36:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:36:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:36:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:36:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:36:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:36:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 88 = '' 2024-09-18T01:36:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:36:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:37:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:37:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:37:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.status 2024-09-18T01:37:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-18T01:37:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=88 2024-09-18T01:37:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 88 = 88 2024-09-18T01:37:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-18T01:37:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:37:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:37:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-18T01:37:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_2': 2024-09-18T01:37:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5857f498 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5857f498 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:28:42 2024 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:28:42 2024 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:28:42 2024 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:37:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:37:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:37:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 9dc75a63-a2d8-4ff1-bf1e-f7d6b7f28a2c 2024-09-18T01:37:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:37:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_2.export 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.export 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_2.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.export 2024-09-18T01:37:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_2.export 2024-09-18T01:37:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.export 2024-09-18T01:37:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_2.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.export 2024-09-18T01:37:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_2.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_2.export 2024-09-18T01:37:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:37:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_3 2024-09-18T01:37:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-18T01:37:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:37:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-18T01:37:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-18T01:37:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:37:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-18T01:37:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-18T01:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:37:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:37:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:37:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:37:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:37:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-18T01:37:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-18T01:37:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-18T01:37:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:37:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:37:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:37:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:37:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-18T01:37:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-18T01:37:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:37:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:37:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:37:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-09-18T01:37:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-18T01:37:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2024-09-18T01:37:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:37:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:37:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:37:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2024-09-18T01:37:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:37:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:37:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:37:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:37:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:37:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:37:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-18T01:37:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_3.status 2024-09-18T01:37:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_3.status 2024-09-18T01:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-18T01:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_3.status 2024-09-18T01:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-18T01:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-18T01:37:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 90 2024-09-18T01:37:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_3.status 2024-09-18T01:37:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_3.status 2024-09-18T01:37:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-18T01:37:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:37:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=90 2024-09-18T01:37:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:37:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-18T01:37:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=37 2024-09-18T01:37:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 90 = 37 2024-09-18T01:37:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:37:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-18T01:37:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 90 = '' 2024-09-18T01:37:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:37:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-18T01:37:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 90 = '' 2024-09-18T01:37:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:37:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.status 2024-09-18T01:37:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-18T01:37:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=90 2024-09-18T01:37:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 90 = 90 2024-09-18T01:37:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-18T01:37:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_3': 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b548456ee6 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b548456ee6 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:37:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:28:53 2024 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:28:53 2024 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:28:53 2024 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 4cad14e9-5f0f-48cb-acf6-96241971c0cd 2024-09-18T01:37:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_3.export 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.export 2024-09-18T01:37:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_3.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.export 2024-09-18T01:37:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_3.export 2024-09-18T01:37:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.export 2024-09-18T01:37:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_3.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.export 2024-09-18T01:37:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_3.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_3.export 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_4 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:37:11.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.44730716213073984) > weight (0.25) 2024-09-18T01:37:11.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.733867960483539) > weight (0.25) 2024-09-18T01:37:11.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8057725542658962) > weight (0.25) 2024-09-18T01:37:11.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4260435948630982) > weight (0.25) 2024-09-18T01:37:11.941 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-18T01:37:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-18T01:37:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:37:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:37:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:37:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-18T01:37:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:37:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:37:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:37:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:37:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-18T01:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-18T01:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:37:13.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:13.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:13.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:13.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-18T01:37:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-18T01:37:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-09-18T01:37:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:37:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:37:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:37:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:37:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:37:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-09-18T01:37:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-18T01:37:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:37:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:37:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:37:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2024-09-18T01:37:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:37:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:37:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:37:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:37:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:37:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-18T01:37:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-18T01:37:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_4.status 2024-09-18T01:37:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_4.status 2024-09-18T01:37:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-18T01:37:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_4.status 2024-09-18T01:37:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-18T01:37:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-18T01:37:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 92 2024-09-18T01:37:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_4.status 2024-09-18T01:37:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_4.status 2024-09-18T01:37:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-18T01:37:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:37:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=92 2024-09-18T01:37:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:37:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-18T01:37:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=38 2024-09-18T01:37:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 92 = 38 2024-09-18T01:37:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:37:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-18T01:37:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 92 = '' 2024-09-18T01:37:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:37:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-18T01:37:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 92 = '' 2024-09-18T01:37:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:37:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.status 2024-09-18T01:37:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-18T01:37:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=92 2024-09-18T01:37:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 92 = 92 2024-09-18T01:37:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-18T01:37:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:37:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-18T01:37:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_4': 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5442eb300 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5442eb300 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:37:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:28:59 2024 2024-09-18T01:37:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:28:59 2024 2024-09-18T01:37:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:28:59 2024 2024-09-18T01:37:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:37:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:37:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:37:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 69e2ede3-6b24-471e-8e85-a7ce3370267a 2024-09-18T01:37:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_4.export 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.export 2024-09-18T01:37:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_4.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.export 2024-09-18T01:37:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_4.export 2024-09-18T01:37:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.export 2024-09-18T01:37:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_4.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.export 2024-09-18T01:37:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_4.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_4.export 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_5 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:37:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-18T01:37:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:37:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:37:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:37:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-18T01:37:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:37:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:37:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:37:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:37:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:37:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:37:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:37:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:37:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:37:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-18T01:37:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-18T01:37:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:37:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:37:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:37:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-18T01:37:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-18T01:37:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:37:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:37:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:37:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-18T01:37:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-18T01:37:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-09-18T01:37:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:37:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:37:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:37:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:37:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-09-18T01:37:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:37:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:37:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:37:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:37:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:37:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:37:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:37:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-18T01:37:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:37:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:37:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-18T01:37:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-18T01:37:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_5.status 2024-09-18T01:37:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_5.status 2024-09-18T01:37:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-18T01:37:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_5.status 2024-09-18T01:37:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-18T01:37:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-18T01:37:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 94 2024-09-18T01:37:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_5.status 2024-09-18T01:37:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_5.status 2024-09-18T01:37:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-18T01:37:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:37:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=94 2024-09-18T01:37:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:37:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-18T01:37:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=39 2024-09-18T01:37:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 94 = 39 2024-09-18T01:37:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:37:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-18T01:37:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 94 = '' 2024-09-18T01:37:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-18T01:37:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 94 = '' 2024-09-18T01:37:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:37:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.status 2024-09-18T01:37:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-18T01:37:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=94 2024-09-18T01:37:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 94 = 94 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:37:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-18T01:37:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_5': 2024-09-18T01:37:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:37:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:37:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:37:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b52b81ae60 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b52b81ae60 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:07 2024 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:07 2024 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:07 2024 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: f65227d5-766c-440e-beb8-127249467502 2024-09-18T01:37:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-18T01:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_5.export 2024-09-18T01:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.export 2024-09-18T01:37:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_5.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.export 2024-09-18T01:37:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_5.export 2024-09-18T01:37:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.export 2024-09-18T01:37:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_5.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.export 2024-09-18T01:37:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_5.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_5.export 2024-09-18T01:37:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:37:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_6 2024-09-18T01:37:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-18T01:37:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:37:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-18T01:37:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:37:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:37:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:37:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-18T01:37:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:37:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:37:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:37:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:37:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:37:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:37:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:37:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:37:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:37:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:37:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:37:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:37:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:37:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:37:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:37:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-18T01:37:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-18T01:37:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:37:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-18T01:37:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-18T01:37:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-09-18T01:37:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:37:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:37:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:37:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:37:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-09-18T01:37:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-18T01:37:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_6 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:37:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:37:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:37:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_6 2024-09-18T01:37:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:37:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:37:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:37:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:37:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-18T01:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_6.status 2024-09-18T01:37:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_6.status 2024-09-18T01:37:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-18T01:37:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_6.status 2024-09-18T01:37:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-18T01:37:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-18T01:37:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 96 2024-09-18T01:37:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_6.status 2024-09-18T01:37:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_6.status 2024-09-18T01:37:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-18T01:37:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:37:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=96 2024-09-18T01:37:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:37:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-18T01:37:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=40 2024-09-18T01:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 96 = 40 2024-09-18T01:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:37:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-18T01:37:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=40 2024-09-18T01:37:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 96 = 40 2024-09-18T01:37:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-18T01:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 96 = '' 2024-09-18T01:37:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:37:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.status 2024-09-18T01:37:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-18T01:37:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=96 2024-09-18T01:37:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 96 = 96 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:37:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-18T01:37:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_6': 2024-09-18T01:37:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:37:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5e5019ce 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5e5019ce 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:16 2024 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:16 2024 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:16 2024 2024-09-18T01:37:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:37:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:37:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:37:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 72ab9ec5-f35c-428f-a831-18f79d59a14f 2024-09-18T01:37:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_6.export 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.export 2024-09-18T01:37:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_6.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.export 2024-09-18T01:37:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_6.export 2024-09-18T01:37:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.export 2024-09-18T01:37:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_6.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.export 2024-09-18T01:37:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_6.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_6.export 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_7 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:37:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-18T01:37:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-18T01:37:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:37:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-18T01:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:37:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-18T01:37:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:37:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:37:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:37:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:37:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:37:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:37:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:37:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:37:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-18T01:37:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:37:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:37:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:37:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:37:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-18T01:37:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-18T01:37:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-09-18T01:37:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:37:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:37:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:37:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:37:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-09-18T01:37:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-18T01:37:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:37:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:37:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:37:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-09-18T01:37:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-18T01:37:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:37:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:37:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2024-09-18T01:37:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:37:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:37:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:37:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:37:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:37:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:37:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-18T01:37:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_7.status 2024-09-18T01:37:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_7.status 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_7.status 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-18T01:37:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 98 2024-09-18T01:37:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_7.status 2024-09-18T01:37:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_7.status 2024-09-18T01:37:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-18T01:37:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:37:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=98 2024-09-18T01:37:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:37:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-18T01:37:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=41 2024-09-18T01:37:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 98 = 41 2024-09-18T01:37:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-18T01:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=41 2024-09-18T01:37:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 98 = 41 2024-09-18T01:37:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:37:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-18T01:37:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 98 = '' 2024-09-18T01:37:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:37:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.status 2024-09-18T01:37:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-18T01:37:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=98 2024-09-18T01:37:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 98 = 98 2024-09-18T01:37:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-18T01:37:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:37:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-18T01:37:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_7': 2024-09-18T01:37:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:37:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:37:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:37:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5ba0d814 2024-09-18T01:37:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5ba0d814 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:29 2024 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:29 2024 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:29 2024 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 4597c48a-8662-495e-a897-2ee7f1408d91 2024-09-18T01:37:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:37:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-18T01:37:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:37:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:37:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_7.export 2024-09-18T01:37:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.export 2024-09-18T01:37:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_7.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.export 2024-09-18T01:37:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_7.export 2024-09-18T01:37:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.export 2024-09-18T01:37:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_7.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.export 2024-09-18T01:37:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_7.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_7.export 2024-09-18T01:37:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:37:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_8 2024-09-18T01:37:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-18T01:37:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-18T01:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-18T01:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-18T01:37:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:37:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:37:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:37:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-18T01:37:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:37:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:37:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:37:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:37:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:37:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:37:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:37:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:37:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-18T01:37:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:37:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:37:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:37:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-18T01:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-18T01:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:37:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:37:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:37:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:37:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-18T01:37:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-18T01:37:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:37:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:37:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:37:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-09-18T01:37:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-18T01:37:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:37:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2024-09-18T01:37:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:37:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:37:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:37:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:37:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:37:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:37:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:37:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-18T01:37:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_8.status 2024-09-18T01:37:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_8.status 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_8.status 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-18T01:37:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 100 2024-09-18T01:37:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_8.status 2024-09-18T01:37:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_8.status 2024-09-18T01:37:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-18T01:37:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:37:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=100 2024-09-18T01:37:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:37:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-18T01:37:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=42 2024-09-18T01:37:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 100 = 42 2024-09-18T01:37:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:37:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-18T01:37:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=42 2024-09-18T01:37:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 100 = 42 2024-09-18T01:37:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:37:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-18T01:37:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 100 = '' 2024-09-18T01:37:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:37:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.status 2024-09-18T01:37:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-18T01:37:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=100 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 100 = 100 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:37:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-18T01:37:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_8': 2024-09-18T01:37:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:37:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:37:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:37:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5ae1db11b 2024-09-18T01:37:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5ae1db11b 2024-09-18T01:37:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:37:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:36 2024 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:36 2024 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:36 2024 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:37:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: b4e5b9d2-efbb-4d6e-b309-ef7dc24ba029 2024-09-18T01:37:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:37:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-18T01:37:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:37:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:37:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_8.export 2024-09-18T01:37:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.export 2024-09-18T01:37:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_8.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.export 2024-09-18T01:37:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_8.export 2024-09-18T01:37:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.export 2024-09-18T01:37:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_8.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.export 2024-09-18T01:37:51.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_8.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_8.export 2024-09-18T01:37:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:37:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_9 2024-09-18T01:37:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-18T01:37:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:37:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-18T01:37:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-18T01:37:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:37:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-18T01:37:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:37:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:37:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:37:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-18T01:37:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:37:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:37:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:37:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:37:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:37:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:37:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:37:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:37:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:37:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:37:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:37:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:37:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:37:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-18T01:37:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-18T01:37:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-09-18T01:37:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:37:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:37:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:37:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:37:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:37:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:37:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-09-18T01:37:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-18T01:37:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2024-09-18T01:37:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:37:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:37:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:37:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:37:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:37:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:37:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2024-09-18T01:37:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-18T01:37:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:37:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:37:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:37:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_9 2024-09-18T01:37:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:37:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:37:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:37:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:37:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:37:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:37:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:37:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:37:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:37:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_9 2024-09-18T01:37:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:37:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:37:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:37:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:37:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:37:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:37:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-18T01:37:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_9.status 2024-09-18T01:37:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_9.status 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_9.status 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-18T01:37:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 102 2024-09-18T01:37:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_9.status 2024-09-18T01:37:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_9.status 2024-09-18T01:37:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-18T01:37:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:37:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=102 2024-09-18T01:37:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:37:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-18T01:37:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=43 2024-09-18T01:37:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 102 = 43 2024-09-18T01:37:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:37:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-18T01:37:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=43 2024-09-18T01:37:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 102 = 43 2024-09-18T01:37:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:37:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-18T01:37:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:37:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:37:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:37:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:37:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:37:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 102 = '' 2024-09-18T01:37:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:37:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:37:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.status 2024-09-18T01:37:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-18T01:37:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:37:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:37:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=102 2024-09-18T01:37:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 102 = 102 2024-09-18T01:37:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-18T01:37:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:37:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:37:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:37:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:37:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-18T01:37:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_9': 2024-09-18T01:37:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:37:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:37:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:37:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b5c0b63e7a 2024-09-18T01:37:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b5c0b63e7a 2024-09-18T01:37:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:37:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:46 2024 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:46 2024 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:46 2024 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 68d70003-8fef-4a72-b907-b819e314282e 2024-09-18T01:37:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:37:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-18T01:37:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:37:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:37:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_9.export 2024-09-18T01:37:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.export 2024-09-18T01:37:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_9.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.export 2024-09-18T01:37:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_9.export 2024-09-18T01:37:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.export 2024-09-18T01:37:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:37:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_9.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.export 2024-09-18T01:37:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_9.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_9.export 2024-09-18T01:37:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:37:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:37:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_10 2024-09-18T01:37:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-18T01:37:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:37:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:37:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:37:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:37:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:38:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-18T01:38:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-18T01:38:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:38:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:38:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:38:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:38:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:38:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:38:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:38:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:38:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:38:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:38:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:38:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:38:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:38:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-18T01:38:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-18T01:38:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-18T01:38:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:38:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:38:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:38:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:38:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-18T01:38:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-18T01:38:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:38:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:38:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-09-18T01:38:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-18T01:38:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:38:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:38:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:38:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2024-09-18T01:38:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:38:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:38:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:38:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:38:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:38:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:38:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:38:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-18T01:38:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:38:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:38:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-18T01:38:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-18T01:38:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_10.status 2024-09-18T01:38:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_10.status 2024-09-18T01:38:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-18T01:38:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_10.status 2024-09-18T01:38:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-18T01:38:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-18T01:38:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 104 2024-09-18T01:38:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_10.status 2024-09-18T01:38:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_10.status 2024-09-18T01:38:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-18T01:38:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:38:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=104 2024-09-18T01:38:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:38:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-18T01:38:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=44 2024-09-18T01:38:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 104 = 44 2024-09-18T01:38:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:38:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-18T01:38:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=44 2024-09-18T01:38:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 104 = 44 2024-09-18T01:38:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-18T01:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:38:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:38:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:38:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:38:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 104 = '' 2024-09-18T01:38:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:38:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.status 2024-09-18T01:38:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-18T01:38:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=104 2024-09-18T01:38:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 104 = 104 2024-09-18T01:38:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-18T01:38:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:38:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-18T01:38:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_10': 2024-09-18T01:38:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:38:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:38:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:38:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6f346abe7 2024-09-18T01:38:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6f346abe7 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:29:53 2024 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:29:53 2024 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:29:53 2024 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: c330e23e-03a4-45ae-9fa0-73ca0a1ac911 2024-09-18T01:38:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:38:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-18T01:38:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:38:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:38:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_10.export 2024-09-18T01:38:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.export 2024-09-18T01:38:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_10.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.export 2024-09-18T01:38:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_10.export 2024-09-18T01:38:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.export 2024-09-18T01:38:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_10.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.export 2024-09-18T01:38:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_10.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_10.export 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_11 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-18T01:38:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:38:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:38:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:38:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:38:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:38:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:38:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:38:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:38:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:38:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-18T01:38:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-18T01:38:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:38:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:38:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:38:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-18T01:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-18T01:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-09-18T01:38:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:38:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:38:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-09-18T01:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-18T01:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-09-18T01:38:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:38:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:38:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-09-18T01:38:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:38:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:38:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:38:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:38:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_11.status 2024-09-18T01:38:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_11.status 2024-09-18T01:38:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-18T01:38:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_11.status 2024-09-18T01:38:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-18T01:38:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-18T01:38:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 106 2024-09-18T01:38:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_11.status 2024-09-18T01:38:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_11.status 2024-09-18T01:38:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-18T01:38:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:38:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=106 2024-09-18T01:38:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:38:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-18T01:38:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=45 2024-09-18T01:38:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 106 = 45 2024-09-18T01:38:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:38:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-18T01:38:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=45 2024-09-18T01:38:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 106 = 45 2024-09-18T01:38:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:38:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-18T01:38:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:38:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:38:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:38:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:38:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 106 = '' 2024-09-18T01:38:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:38:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.status 2024-09-18T01:38:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-18T01:38:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=106 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 106 = 106 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:38:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-18T01:38:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_11': 2024-09-18T01:38:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:38:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b618177187 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b618177187 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:02 2024 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:02 2024 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:02 2024 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:38:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:38:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: f4d6f564-c302-4459-b459-852582eb1804 2024-09-18T01:38:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_11.export 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.export 2024-09-18T01:38:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_11.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.export 2024-09-18T01:38:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_11.export 2024-09-18T01:38:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.export 2024-09-18T01:38:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_11.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.export 2024-09-18T01:38:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_11.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_11.export 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_12 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:38:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-18T01:38:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-18T01:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:17.942 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6262144095671404) > weight (0.25) 2024-09-18T01:38:17.942 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.868630638550232) > weight (0.25) 2024-09-18T01:38:17.942 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7153599671225985) > weight (0.25) 2024-09-18T01:38:17.942 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-18T01:38:17.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-18T01:38:17.943 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-18T01:38:17.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:38:17.942+0000 7fc10c977640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 45868) UID: 1000 2024-09-18T01:38:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-18T01:38:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:38:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:38:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-18T01:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:38:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:38:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:38:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:38:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-18T01:38:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:38:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:38:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:38:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:38:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:38:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:38:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-18T01:38:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-18T01:38:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:38:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:38:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:38:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-18T01:38:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-18T01:38:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:38:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:38:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:38:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-09-18T01:38:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-18T01:38:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2024-09-18T01:38:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:38:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:38:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:38:18.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:38:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:38:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2024-09-18T01:38:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:38:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:38:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:38:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:38:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:38:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:38:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:38:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-18T01:38:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_12.status 2024-09-18T01:38:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_12.status 2024-09-18T01:38:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-18T01:38:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_12.status 2024-09-18T01:38:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-18T01:38:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-18T01:38:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 108 2024-09-18T01:38:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_12.status 2024-09-18T01:38:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_12.status 2024-09-18T01:38:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-18T01:38:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:38:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=108 2024-09-18T01:38:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:38:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-18T01:38:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=46 2024-09-18T01:38:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 108 = 46 2024-09-18T01:38:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:38:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-18T01:38:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=46 2024-09-18T01:38:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 108 = 46 2024-09-18T01:38:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:38:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-18T01:38:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:38:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:38:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:38:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:38:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 108 = '' 2024-09-18T01:38:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:38:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.status 2024-09-18T01:38:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-18T01:38:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=108 2024-09-18T01:38:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 108 = 108 2024-09-18T01:38:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-18T01:38:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:38:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-18T01:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_12': 2024-09-18T01:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b629017bb0 2024-09-18T01:38:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b629017bb0 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:08 2024 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:08 2024 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:08 2024 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: d652dfd4-0db3-409b-a60f-853dfb4b8bbf 2024-09-18T01:38:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-18T01:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_12.export 2024-09-18T01:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.export 2024-09-18T01:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_12.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.export 2024-09-18T01:38:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_12.export 2024-09-18T01:38:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.export 2024-09-18T01:38:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_12.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.export 2024-09-18T01:38:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_12.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_12.export 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_13 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-18T01:38:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:38:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:38:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:38:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-18T01:38:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:38:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:38:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:38:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:38:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:38:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:38:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:38:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:38:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:38:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:38:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-18T01:38:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-18T01:38:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-09-18T01:38:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:38:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:38:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:38:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:38:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:38:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:38:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-09-18T01:38:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-18T01:38:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:38:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:38:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:38:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2024-09-18T01:38:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-18T01:38:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2024-09-18T01:38:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:38:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:38:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:38:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:38:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2024-09-18T01:38:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:38:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:38:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:38:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:38:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:38:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-18T01:38:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_13.status 2024-09-18T01:38:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_13.status 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_13.status 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-18T01:38:26.943 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-18T01:38:26.943 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:38:26.943 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-18T01:38:26.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-18T01:38:26.944 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-18T01:38:26.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-18T01:38:26.947 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-09-18T01:38:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 110 2024-09-18T01:38:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_13.status 2024-09-18T01:38:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_13.status 2024-09-18T01:38:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-18T01:38:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:38:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=110 2024-09-18T01:38:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:38:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-18T01:38:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=47 2024-09-18T01:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 110 = 47 2024-09-18T01:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:38:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-18T01:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=47 2024-09-18T01:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 110 = 47 2024-09-18T01:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-18T01:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:38:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:38:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:38:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:38:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 110 = '' 2024-09-18T01:38:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:38:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.status 2024-09-18T01:38:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-18T01:38:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=110 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 110 = 110 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:38:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-18T01:38:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_13': 2024-09-18T01:38:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:38:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b68c4804f8 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b68c4804f8 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:14 2024 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:14 2024 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:14 2024 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:38:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:38:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 0b20a1f2-779a-42c1-bde9-cc45daa1eb9b 2024-09-18T01:38:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:38:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-18T01:38:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:38:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:38:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_13.export 2024-09-18T01:38:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.export 2024-09-18T01:38:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_13.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.export 2024-09-18T01:38:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_13.export 2024-09-18T01:38:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.export 2024-09-18T01:38:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_13.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.export 2024-09-18T01:38:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_13.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_13.export 2024-09-18T01:38:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:38:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_14 2024-09-18T01:38:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-18T01:38:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:38:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-18T01:38:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:38:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:38:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:38:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-18T01:38:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:38:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:38:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:38:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:38:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:38:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:38:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:38:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:38:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-18T01:38:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-18T01:38:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:38:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:38:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:38:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-18T01:38:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-18T01:38:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:38:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-09-18T01:38:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-18T01:38:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:38:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:38:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:38:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:38:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2024-09-18T01:38:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:38:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:38:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:38:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:38:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-18T01:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-18T01:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_14.status 2024-09-18T01:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_14.status 2024-09-18T01:38:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-18T01:38:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_14.status 2024-09-18T01:38:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-18T01:38:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-18T01:38:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 112 2024-09-18T01:38:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_14.status 2024-09-18T01:38:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_14.status 2024-09-18T01:38:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-18T01:38:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:38:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=112 2024-09-18T01:38:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:38:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-18T01:38:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=48 2024-09-18T01:38:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 112 = 48 2024-09-18T01:38:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:38:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-18T01:38:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:38:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:38:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:38:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:38:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 112 = '' 2024-09-18T01:38:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:38:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-18T01:38:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:38:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:38:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:38:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:38:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 112 = '' 2024-09-18T01:38:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:38:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.status 2024-09-18T01:38:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-18T01:38:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=112 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 112 = 112 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:38:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-18T01:38:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_14': 2024-09-18T01:38:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:38:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:38:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:38:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b67e5db43d 2024-09-18T01:38:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b67e5db43d 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:26 2024 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:26 2024 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:26 2024 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3c45b3f0-662d-4182-8d89-837d0b48bc1a 2024-09-18T01:38:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:38:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-18T01:38:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:38:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:38:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_14.export 2024-09-18T01:38:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.export 2024-09-18T01:38:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_14.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.export 2024-09-18T01:38:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_14.export 2024-09-18T01:38:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.export 2024-09-18T01:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_14.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.export 2024-09-18T01:38:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_14.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_14.export 2024-09-18T01:38:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:38:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_15 2024-09-18T01:38:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-18T01:38:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:38:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:41.742 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6355839950554918) > weight (0.25) 2024-09-18T01:38:41.742 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6841665406199606) > weight (0.25) 2024-09-18T01:38:41.742 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4154699011501809) > weight (0.25) 2024-09-18T01:38:41.742 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7650654691484606) > weight (0.25) 2024-09-18T01:38:41.742 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-09-18T01:38:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-18T01:38:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:38:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:38:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:38:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-18T01:38:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:38:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:38:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-18T01:38:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:38:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:38:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:38:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:38:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:38:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:38:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:38:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:38:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-18T01:38:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-18T01:38:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:38:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:38:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:38:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:38:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-18T01:38:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-18T01:38:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-09-18T01:38:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:38:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:38:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:38:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:38:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:38:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-09-18T01:38:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-18T01:38:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_15 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:38:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:38:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_15 2024-09-18T01:38:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:38:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:38:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:38:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:38:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:38:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:38:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:38:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-18T01:38:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-18T01:38:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_15.status 2024-09-18T01:38:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_15.status 2024-09-18T01:38:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-18T01:38:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_15.status 2024-09-18T01:38:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-18T01:38:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-18T01:38:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 114 2024-09-18T01:38:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_15.status 2024-09-18T01:38:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_15.status 2024-09-18T01:38:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-18T01:38:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:38:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=114 2024-09-18T01:38:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:38:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-18T01:38:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=49 2024-09-18T01:38:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 114 = 49 2024-09-18T01:38:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:38:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-18T01:38:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=49 2024-09-18T01:38:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 114 = 49 2024-09-18T01:38:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:38:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-18T01:38:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:38:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:38:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:38:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:38:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 114 = '' 2024-09-18T01:38:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:38:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.status 2024-09-18T01:38:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-18T01:38:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=114 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 114 = 114 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:38:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-18T01:38:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_15': 2024-09-18T01:38:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6efda520f 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6efda520f 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:37 2024 2024-09-18T01:38:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:37 2024 2024-09-18T01:38:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:37 2024 2024-09-18T01:38:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:38:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:38:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:38:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 15b04f61-470a-4fa7-811f-b4270495f8f7 2024-09-18T01:38:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_15.export 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.export 2024-09-18T01:38:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_15.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.export 2024-09-18T01:38:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_15.export 2024-09-18T01:38:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.export 2024-09-18T01:38:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_15.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.export 2024-09-18T01:38:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_15.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_15.export 2024-09-18T01:38:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:38:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_16 2024-09-18T01:38:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-18T01:38:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:38:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-18T01:38:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-18T01:38:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:38:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:38:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:38:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:38:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:38:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:38:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:38:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:38:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-18T01:38:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-18T01:38:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-18T01:38:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:38:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:38:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:38:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-18T01:38:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-18T01:38:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:38:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:38:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:38:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-18T01:38:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-18T01:38:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-09-18T01:38:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:38:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:38:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:38:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:38:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-09-18T01:38:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:38:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:38:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:38:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:38:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:38:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-18T01:38:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-18T01:38:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_16.status 2024-09-18T01:38:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_16.status 2024-09-18T01:38:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-18T01:38:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_16.status 2024-09-18T01:38:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-18T01:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-18T01:38:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 116 2024-09-18T01:38:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_16.status 2024-09-18T01:38:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_16.status 2024-09-18T01:38:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-18T01:38:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:38:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=116 2024-09-18T01:38:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:38:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-18T01:38:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=50 2024-09-18T01:38:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 116 = 50 2024-09-18T01:38:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:38:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-18T01:38:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=50 2024-09-18T01:38:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 116 = 50 2024-09-18T01:38:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:38:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-18T01:38:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:38:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:38:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:38:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:38:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 116 = '' 2024-09-18T01:38:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:38:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.status 2024-09-18T01:38:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-18T01:38:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=116 2024-09-18T01:38:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 116 = 116 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:38:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-18T01:38:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_16': 2024-09-18T01:38:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b689b7fed0 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b689b7fed0 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:44 2024 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:44 2024 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:44 2024 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:38:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 0a9c0c62-9c63-4428-abf9-46d978cda72b 2024-09-18T01:38:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:38:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-18T01:38:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_16.export 2024-09-18T01:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.export 2024-09-18T01:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_16.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.export 2024-09-18T01:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_16.export 2024-09-18T01:38:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.export 2024-09-18T01:38:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:38:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_16.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.export 2024-09-18T01:38:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_16.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_16.export 2024-09-18T01:38:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:38:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_17 2024-09-18T01:38:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-18T01:38:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:38:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:38:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:38:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-18T01:38:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:38:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:38:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:38:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:38:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-18T01:38:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:38:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:38:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:38:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:38:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:38:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:38:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:38:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:38:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-18T01:38:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-18T01:38:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:38:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:38:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:38:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:38:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:38:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:38:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-18T01:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-18T01:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-09-18T01:38:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:38:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:38:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:38:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:38:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:38:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-09-18T01:38:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-18T01:38:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:38:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:38:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:38:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2024-09-18T01:38:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:38:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:38:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:38:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:38:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2024-09-18T01:38:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:38:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:38:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:38:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:38:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:38:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:38:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:38:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:38:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-18T01:38:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:38:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:38:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-18T01:38:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:38:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-18T01:38:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_17.status 2024-09-18T01:38:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_17.status 2024-09-18T01:38:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-18T01:38:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_17.status 2024-09-18T01:38:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:38:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:38:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-18T01:38:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-18T01:38:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 118 2024-09-18T01:38:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_17.status 2024-09-18T01:38:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:38:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_17.status 2024-09-18T01:38:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-18T01:38:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:38:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=118 2024-09-18T01:38:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:38:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:38:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:38:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:38:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-18T01:38:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=51 2024-09-18T01:38:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 118 = 51 2024-09-18T01:38:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-18T01:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:38:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:38:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=51 2024-09-18T01:38:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 118 = 51 2024-09-18T01:38:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:38:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:39:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:39:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:39:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:39:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-18T01:39:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 118 = '' 2024-09-18T01:39:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:39:02.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:39:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:39:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.status 2024-09-18T01:39:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-18T01:39:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=118 2024-09-18T01:39:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 118 = 118 2024-09-18T01:39:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-18T01:39:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:39:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:39:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-18T01:39:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_17': 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6bf5fc62b 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6bf5fc62b 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:50 2024 2024-09-18T01:39:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:50 2024 2024-09-18T01:39:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:50 2024 2024-09-18T01:39:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:39:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:39:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:39:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 2af45ec2-1273-4f89-9d4a-659567f6c7cc 2024-09-18T01:39:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_17.export 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.export 2024-09-18T01:39:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_17.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.export 2024-09-18T01:39:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_17.export 2024-09-18T01:39:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.export 2024-09-18T01:39:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_17.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.export 2024-09-18T01:39:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_17.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_17.export 2024-09-18T01:39:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:39:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_18 2024-09-18T01:39:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-18T01:39:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:39:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-18T01:39:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-18T01:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:39:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-18T01:39:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:39:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:39:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-18T01:39:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:39:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:39:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:39:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:39:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:39:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:39:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:39:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-18T01:39:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-18T01:39:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:39:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:39:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:39:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-18T01:39:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-18T01:39:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-09-18T01:39:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:39:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:39:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:39:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:39:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-09-18T01:39:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-18T01:39:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2024-09-18T01:39:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:39:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:39:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:39:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:39:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:39:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:39:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2024-09-18T01:39:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:39:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:39:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:39:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:39:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:39:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:39:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:39:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_18.status 2024-09-18T01:39:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_18.status 2024-09-18T01:39:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-18T01:39:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_18.status 2024-09-18T01:39:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-18T01:39:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-18T01:39:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 120 2024-09-18T01:39:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_18.status 2024-09-18T01:39:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_18.status 2024-09-18T01:39:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-18T01:39:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:39:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=120 2024-09-18T01:39:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:39:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-18T01:39:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=52 2024-09-18T01:39:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 120 = 52 2024-09-18T01:39:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:39:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-18T01:39:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=52 2024-09-18T01:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 120 = 52 2024-09-18T01:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-18T01:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 120 = '' 2024-09-18T01:39:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:39:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.status 2024-09-18T01:39:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-18T01:39:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=120 2024-09-18T01:39:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 120 = 120 2024-09-18T01:39:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-18T01:39:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:39:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-18T01:39:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_18': 2024-09-18T01:39:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:39:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:39:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:39:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b61809d4a 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b61809d4a 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:30:58 2024 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:30:58 2024 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:30:58 2024 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: e28fa577-03c0-4443-8c02-6edcbe8d5994 2024-09-18T01:39:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:39:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-18T01:39:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:39:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:39:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_18.export 2024-09-18T01:39:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.export 2024-09-18T01:39:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_18.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.export 2024-09-18T01:39:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_18.export 2024-09-18T01:39:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.export 2024-09-18T01:39:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_18.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.export 2024-09-18T01:39:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_18.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_18.export 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_19 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:39:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-18T01:39:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-18T01:39:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:39:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:39:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:39:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-18T01:39:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:39:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:39:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:39:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:39:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:39:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:39:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:39:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:39:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-18T01:39:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-18T01:39:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-09-18T01:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:39:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:39:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-09-18T01:39:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-18T01:39:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:39:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:39:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:39:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2024-09-18T01:39:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-18T01:39:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_19 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:39:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:39:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:39:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_19 2024-09-18T01:39:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:39:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:39:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:39:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:39:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:39:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:39:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:39:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-18T01:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_19.status 2024-09-18T01:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_19.status 2024-09-18T01:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-18T01:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_19.status 2024-09-18T01:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-18T01:39:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-18T01:39:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 122 2024-09-18T01:39:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_19.status 2024-09-18T01:39:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_19.status 2024-09-18T01:39:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-18T01:39:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:39:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=122 2024-09-18T01:39:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:39:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-18T01:39:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=53 2024-09-18T01:39:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 122 = 53 2024-09-18T01:39:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:39:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-18T01:39:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 122 = '' 2024-09-18T01:39:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:39:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-18T01:39:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 122 = '' 2024-09-18T01:39:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:39:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.status 2024-09-18T01:39:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-18T01:39:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=122 2024-09-18T01:39:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 122 = 122 2024-09-18T01:39:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-18T01:39:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:39:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-18T01:39:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_19': 2024-09-18T01:39:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:39:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:39:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:39:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b67d798ad7 2024-09-18T01:39:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b67d798ad7 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:04 2024 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:04 2024 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:04 2024 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: fc033df2-2782-4b60-b444-9993613dbf5d 2024-09-18T01:39:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:39:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-18T01:39:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:39:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:39:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_19.export 2024-09-18T01:39:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.export 2024-09-18T01:39:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_19.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.export 2024-09-18T01:39:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_19.export 2024-09-18T01:39:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.export 2024-09-18T01:39:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_19.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.export 2024-09-18T01:39:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_19.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_19.export 2024-09-18T01:39:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:39:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_20 2024-09-18T01:39:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-18T01:39:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:39:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-18T01:39:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:39:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:39:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:39:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-18T01:39:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:39:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:39:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:39:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:39:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:39:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:39:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:39:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:39:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:39:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:39:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:39:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-18T01:39:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-18T01:39:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:39:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:39:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:39:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-18T01:39:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-18T01:39:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:39:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:39:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:39:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-09-18T01:39:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-18T01:39:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:39:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:39:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:39:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-09-18T01:39:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:39:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:39:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:39:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:39:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-18T01:39:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-18T01:39:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_20.status 2024-09-18T01:39:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_20.status 2024-09-18T01:39:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-18T01:39:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_20.status 2024-09-18T01:39:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-18T01:39:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-18T01:39:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 124 2024-09-18T01:39:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_20.status 2024-09-18T01:39:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_20.status 2024-09-18T01:39:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-18T01:39:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:39:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=124 2024-09-18T01:39:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:39:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-18T01:39:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=54 2024-09-18T01:39:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 124 = 54 2024-09-18T01:39:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:39:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-18T01:39:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=54 2024-09-18T01:39:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 124 = 54 2024-09-18T01:39:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:39:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-18T01:39:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 124 = '' 2024-09-18T01:39:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:39:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.status 2024-09-18T01:39:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-18T01:39:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=124 2024-09-18T01:39:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 124 = 124 2024-09-18T01:39:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-18T01:39:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:39:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-18T01:39:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_20': 2024-09-18T01:39:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:39:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:39:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:39:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b61171a814 2024-09-18T01:39:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b61171a814 2024-09-18T01:39:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:39:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:39:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:39:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:39:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:19 2024 2024-09-18T01:39:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:19 2024 2024-09-18T01:39:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:19 2024 2024-09-18T01:39:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:39:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:39:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:39:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: b7c93f86-7ae8-4602-a345-dc61cce15b17 2024-09-18T01:39:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:39:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-18T01:39:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:39:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:39:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_20.export 2024-09-18T01:39:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.export 2024-09-18T01:39:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_20.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.export 2024-09-18T01:39:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_20.export 2024-09-18T01:39:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.export 2024-09-18T01:39:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_20.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.export 2024-09-18T01:39:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_20.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_20.export 2024-09-18T01:39:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:39:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_21 2024-09-18T01:39:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-18T01:39:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:39:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-18T01:39:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-18T01:39:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:39:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-18T01:39:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:39:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:39:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:39:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-18T01:39:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:39:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:39:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:39:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:39:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:39:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:39:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:39:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:39:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-18T01:39:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:39:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:39:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:39:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:39:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:39:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:39:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:39:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:39:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:39:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:39:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-18T01:39:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-18T01:39:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:39:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:39:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:39:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-09-18T01:39:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-18T01:39:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:39:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:39:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:39:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-09-18T01:39:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-18T01:39:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_21 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:39:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:39:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:39:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_21 2024-09-18T01:39:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:39:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:39:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:39:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:39:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:39:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:39:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:39:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-18T01:39:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:39:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:39:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-18T01:39:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-18T01:39:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_21.status 2024-09-18T01:39:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_21.status 2024-09-18T01:39:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-18T01:39:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_21.status 2024-09-18T01:39:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.status 2024-09-18T01:39:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-18T01:39:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-18T01:39:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 126 2024-09-18T01:39:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_21.status 2024-09-18T01:39:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_21.status 2024-09-18T01:39:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-18T01:39:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:39:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=126 2024-09-18T01:39:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:39:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.status 2024-09-18T01:39:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.status 2024-09-18T01:39:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-18T01:39:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 126 = '' 2024-09-18T01:39:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:39:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.status 2024-09-18T01:39:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.status 2024-09-18T01:39:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-18T01:39:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 126 = '' 2024-09-18T01:39:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.status 2024-09-18T01:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.status 2024-09-18T01:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-18T01:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=126 2024-09-18T01:39:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 126 = 126 2024-09-18T01:39:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-18T01:39:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:39:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-18T01:39:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_21': 2024-09-18T01:39:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:39:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:39:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:39:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6ca58d402 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6ca58d402 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:27 2024 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:27 2024 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:27 2024 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 854098b9-42e8-43d8-b6c2-a45666ee66c5 2024-09-18T01:39:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-18T01:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:39:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:39:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_21.export 2024-09-18T01:39:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.export 2024-09-18T01:39:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_21.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.export 2024-09-18T01:39:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_21.export 2024-09-18T01:39:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.export 2024-09-18T01:39:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_21.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.export 2024-09-18T01:39:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_21.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_21.export 2024-09-18T01:39:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:39:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_22 2024-09-18T01:39:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-18T01:39:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:39:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-18T01:39:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:39:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:39:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:39:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:39:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:39:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:39:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:39:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:39:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-18T01:39:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-18T01:39:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:39:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:39:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:39:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-18T01:39:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-18T01:39:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-09-18T01:39:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:39:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:39:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:39:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:39:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:39:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-09-18T01:39:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-18T01:39:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2024-09-18T01:39:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:39:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:39:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:39:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:39:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:39:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:39:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2024-09-18T01:39:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:39:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:39:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:39:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:39:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:39:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:39:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:39:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-18T01:39:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_22.status 2024-09-18T01:39:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_22.status 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_22.status 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.status 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-18T01:39:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 128 2024-09-18T01:39:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_22.status 2024-09-18T01:39:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_22.status 2024-09-18T01:39:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-18T01:39:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:39:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=128 2024-09-18T01:39:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:39:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.status 2024-09-18T01:39:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.status 2024-09-18T01:39:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-18T01:39:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 128 = '' 2024-09-18T01:39:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:39:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.status 2024-09-18T01:39:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.status 2024-09-18T01:39:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-18T01:39:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 128 = '' 2024-09-18T01:39:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:39:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.status 2024-09-18T01:39:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.status 2024-09-18T01:39:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-18T01:39:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=128 2024-09-18T01:39:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 128 = 128 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:39:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_22': 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6e99a45d2 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6e99a45d2 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:39:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:42 2024 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:42 2024 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:42 2024 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: b41be30f-1770-49fc-9175-aad90b2449de 2024-09-18T01:39:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_22.export 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.export 2024-09-18T01:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_22.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.export 2024-09-18T01:39:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_22.export 2024-09-18T01:39:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.export 2024-09-18T01:39:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_22.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.export 2024-09-18T01:39:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_22.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_22.export 2024-09-18T01:39:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_23 2024-09-18T01:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-18T01:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:39:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-18T01:39:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-18T01:39:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:39:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-18T01:39:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:39:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:39:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:39:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:39:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:39:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:39:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:39:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:39:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:39:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:39:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-18T01:39:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-18T01:39:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-09-18T01:39:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:39:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:39:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:39:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:39:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-09-18T01:39:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-18T01:39:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:39:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:39:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:39:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-09-18T01:39:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-18T01:39:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-09-18T01:39:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:39:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:39:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:39:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:39:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-09-18T01:39:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:39:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:39:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:39:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:39:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:39:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:39:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:39:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-18T01:39:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_23.status 2024-09-18T01:39:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_23.status 2024-09-18T01:39:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-18T01:39:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_23.status 2024-09-18T01:39:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-18T01:39:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-18T01:39:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 130 2024-09-18T01:39:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_23.status 2024-09-18T01:39:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_23.status 2024-09-18T01:39:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-18T01:39:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:39:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=130 2024-09-18T01:39:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:39:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-18T01:39:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=57 2024-09-18T01:39:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 130 = 57 2024-09-18T01:39:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:39:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-18T01:39:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 130 = '' 2024-09-18T01:39:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:39:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-18T01:39:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 130 = '' 2024-09-18T01:39:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:39:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.status 2024-09-18T01:39:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-18T01:39:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=130 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 130 = 130 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:39:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-18T01:39:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_23': 2024-09-18T01:39:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:39:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:39:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:39:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b66843ab2d 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b66843ab2d 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:50 2024 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:50 2024 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:50 2024 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 0b0b9342-5291-4bb5-ade8-4fa8d9ff735c 2024-09-18T01:39:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_23.export 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.export 2024-09-18T01:39:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_23.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.export 2024-09-18T01:39:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_23.export 2024-09-18T01:39:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.export 2024-09-18T01:39:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_23.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.export 2024-09-18T01:39:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_23.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_23.export 2024-09-18T01:39:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_24 2024-09-18T01:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-18T01:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-18T01:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-18T01:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:39:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-18T01:39:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:39:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:39:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:39:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:39:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:39:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:39:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:39:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:39:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:39:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-18T01:39:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-18T01:39:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:39:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:39:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:39:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:39:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-18T01:39:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-18T01:39:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:39:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:39:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:39:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:39:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:39:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:39:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-09-18T01:39:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-18T01:39:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:39:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:39:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:39:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:39:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:39:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:39:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:39:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:39:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:39:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2024-09-18T01:39:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:39:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:39:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:39:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:39:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:39:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:39:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:39:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-18T01:39:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_24.status 2024-09-18T01:39:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_24.status 2024-09-18T01:39:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-18T01:39:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_24.status 2024-09-18T01:39:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:39:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-18T01:39:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-18T01:39:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 132 2024-09-18T01:39:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_24.status 2024-09-18T01:39:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_24.status 2024-09-18T01:39:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-18T01:39:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:39:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=132 2024-09-18T01:39:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:39:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-18T01:39:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=58 2024-09-18T01:39:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 132 = 58 2024-09-18T01:39:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:39:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-18T01:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=58 2024-09-18T01:39:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 132 = 58 2024-09-18T01:39:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:39:54.947 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8314151079496148) > weight (0.25) 2024-09-18T01:39:54.948 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8125729316225614) > weight (0.25) 2024-09-18T01:39:54.948 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.86838025022498) > weight (0.25) 2024-09-18T01:39:54.948 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8113021661597646) > weight (0.25) 2024-09-18T01:39:54.948 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-09-18T01:39:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-18T01:39:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:39:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:39:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:39:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:39:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:39:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 132 = '' 2024-09-18T01:39:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:39:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:39:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.status 2024-09-18T01:39:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-18T01:39:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:39:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:39:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=132 2024-09-18T01:39:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 132 = 132 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:39:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:39:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_24': 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6288a0ecf 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6288a0ecf 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:31:59 2024 2024-09-18T01:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:31:59 2024 2024-09-18T01:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:31:59 2024 2024-09-18T01:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 6425ebaf-56a7-47a1-9547-1c0136cbba11 2024-09-18T01:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_24.export 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.export 2024-09-18T01:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_24.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.export 2024-09-18T01:39:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_24.export 2024-09-18T01:39:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.export 2024-09-18T01:39:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:39:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_24.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.export 2024-09-18T01:39:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_24.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_24.export 2024-09-18T01:39:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:39:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:39:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_25 2024-09-18T01:39:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-18T01:39:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:39:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:39:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:39:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:39:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:39:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-18T01:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:40:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:40:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-18T01:40:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:40:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:40:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:40:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:40:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:40:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:40:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:40:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:40:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-18T01:40:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:40:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:40:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:40:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:40:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:40:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:40:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:40:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:40:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-18T01:40:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-18T01:40:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:40:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:40:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-18T01:40:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-18T01:40:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:40:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:40:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:40:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:40:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-09-18T01:40:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-18T01:40:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:40:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2024-09-18T01:40:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:40:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:40:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:40:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:40:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-18T01:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-18T01:40:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_25.status 2024-09-18T01:40:01.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_25.status 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_25.status 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-18T01:40:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 134 2024-09-18T01:40:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_25.status 2024-09-18T01:40:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_25.status 2024-09-18T01:40:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-18T01:40:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:40:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=134 2024-09-18T01:40:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:40:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-18T01:40:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=59 2024-09-18T01:40:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 134 = 59 2024-09-18T01:40:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:40:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-18T01:40:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=59 2024-09-18T01:40:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 134 = 59 2024-09-18T01:40:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:40:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-18T01:40:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 134 = '' 2024-09-18T01:40:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:40:03.795 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:40:03.793+0000 7f78fa24b640 -1 reset not still connected to 0x55b9c8db69c0 2024-09-18T01:40:03.798 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:40:03.798+0000 7f78fa24b640 -1 reset not still connected to 0x55b9d275fba0 2024-09-18T01:40:03.803 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:40:03.802+0000 7f78fa24b640 -1 reset not still connected to 0x55b9dcdd3790 2024-09-18T01:40:03.803 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:40:03.802+0000 7f78fa24b640 -1 reset not still connected to 0x55b9dd3f2000 2024-09-18T01:40:03.807 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:40:03.807+0000 7f78fa24b640 -1 reset not still connected to 0x55b9c8db69c0 2024-09-18T01:40:03.807 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:40:03.807+0000 7f78fa24b640 -1 reset not still connected to 0x55b9d275fba0 2024-09-18T01:40:03.807 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:40:03.807+0000 7f78fa24b640 -1 reset not still connected to 0x55b9dcdd3790 2024-09-18T01:40:03.808 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:40:03.807+0000 7f78fa24b640 -1 reset not still connected to 0x55b9dd3f2000 2024-09-18T01:40:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.status 2024-09-18T01:40:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-18T01:40:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=134 2024-09-18T01:40:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 134 = 134 2024-09-18T01:40:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-18T01:40:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:40:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_25': 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6446a0ccd 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6446a0ccd 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:40:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:40:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:14 2024 2024-09-18T01:40:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:14 2024 2024-09-18T01:40:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:14 2024 2024-09-18T01:40:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:40:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:40:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:40:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 0ee8e883-4767-4c92-9030-1bb0af68068c 2024-09-18T01:40:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_25.export 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.export 2024-09-18T01:40:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_25.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.export 2024-09-18T01:40:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_25.export 2024-09-18T01:40:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.export 2024-09-18T01:40:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_25.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.export 2024-09-18T01:40:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_25.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_25.export 2024-09-18T01:40:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:40:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_26 2024-09-18T01:40:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-18T01:40:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:40:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-18T01:40:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:40:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:40:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:40:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:40:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:40:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:40:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-18T01:40:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:40:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:40:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:40:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-18T01:40:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-18T01:40:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:40:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:40:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:40:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:40:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-18T01:40:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-18T01:40:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-18T01:40:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:40:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:40:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:40:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:40:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-18T01:40:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-18T01:40:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:40:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:40:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:40:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-09-18T01:40:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:40:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:40:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:40:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:40:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:40:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:40:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:40:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-18T01:40:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_26.status 2024-09-18T01:40:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_26.status 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_26.status 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-18T01:40:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 136 2024-09-18T01:40:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_26.status 2024-09-18T01:40:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_26.status 2024-09-18T01:40:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-18T01:40:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:40:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=136 2024-09-18T01:40:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:40:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-18T01:40:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=60 2024-09-18T01:40:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 136 = 60 2024-09-18T01:40:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:40:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-18T01:40:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=60 2024-09-18T01:40:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 136 = 60 2024-09-18T01:40:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:40:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-18T01:40:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 136 = '' 2024-09-18T01:40:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.status 2024-09-18T01:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-18T01:40:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=136 2024-09-18T01:40:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 136 = 136 2024-09-18T01:40:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-18T01:40:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:40:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-18T01:40:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_26': 2024-09-18T01:40:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:40:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:40:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6649db685 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6649db685 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:21 2024 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:21 2024 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:21 2024 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 41d5e821-71ed-41c3-afd1-9e875de13804 2024-09-18T01:40:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:40:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-18T01:40:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:40:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:40:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_26.export 2024-09-18T01:40:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.export 2024-09-18T01:40:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_26.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.export 2024-09-18T01:40:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_26.export 2024-09-18T01:40:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.export 2024-09-18T01:40:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_26.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.export 2024-09-18T01:40:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_26.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_26.export 2024-09-18T01:40:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:40:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_27 2024-09-18T01:40:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-18T01:40:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:40:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-18T01:40:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-18T01:40:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:40:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:40:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:40:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:40:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:40:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:40:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:40:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-18T01:40:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-18T01:40:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-18T01:40:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:40:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:40:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:40:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-18T01:40:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-18T01:40:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-18T01:40:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:40:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:40:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:40:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:40:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:40:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-18T01:40:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-18T01:40:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:40:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:40:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:40:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-09-18T01:40:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:40:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:40:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:40:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:40:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:40:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-18T01:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-18T01:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_27.status 2024-09-18T01:40:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_27.status 2024-09-18T01:40:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-18T01:40:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_27.status 2024-09-18T01:40:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-18T01:40:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-18T01:40:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 138 2024-09-18T01:40:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_27.status 2024-09-18T01:40:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_27.status 2024-09-18T01:40:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-18T01:40:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:40:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=138 2024-09-18T01:40:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:40:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-18T01:40:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=61 2024-09-18T01:40:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 138 = 61 2024-09-18T01:40:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:40:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-18T01:40:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=61 2024-09-18T01:40:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 138 = 61 2024-09-18T01:40:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:40:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-18T01:40:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 138 = '' 2024-09-18T01:40:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:40:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.status 2024-09-18T01:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-18T01:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=138 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 138 = 138 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:40:21.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-18T01:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_27': 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6e033f292 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6e033f292 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:35 2024 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:35 2024 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:35 2024 2024-09-18T01:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: ba4cfda3-b609-4e89-9671-ed59bd54d468 2024-09-18T01:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_27.export 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.export 2024-09-18T01:40:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_27.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.export 2024-09-18T01:40:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_27.export 2024-09-18T01:40:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.export 2024-09-18T01:40:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_27.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.export 2024-09-18T01:40:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_27.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_27.export 2024-09-18T01:40:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:40:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_28 2024-09-18T01:40:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-18T01:40:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:40:23.989 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:23.987+0000 7fadd68ba640 -1 reset not still connected to 0x55d6c3c51450 2024-09-18T01:40:23.990 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:23.987+0000 7fadd68ba640 -1 reset not still connected to 0x55d6c4586340 2024-09-18T01:40:23.990 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:23.987+0000 7fadd68ba640 -1 reset not still connected to 0x55d6c88e6f70 2024-09-18T01:40:24.003 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:24.002+0000 7fadd68ba640 -1 reset not still connected to 0x55d6c95cf6c0 2024-09-18T01:40:24.003 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:24.002+0000 7fadd68ba640 -1 reset not still connected to 0x55d6ca0a7ba0 2024-09-18T01:40:24.003 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:24.002+0000 7fadd68ba640 -1 reset not still connected to 0x55d6ca995040 2024-09-18T01:40:24.003 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:24.002+0000 7fadd68ba640 -1 reset not still connected to 0x55d6cb99a8f0 2024-09-18T01:40:24.003 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:24.002+0000 7fadd68ba640 -1 reset not still connected to 0x55d6cc546f70 2024-09-18T01:40:24.003 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:24.002+0000 7fadd68ba640 -1 reset not still connected to 0x55d6cc5c81a0 2024-09-18T01:40:24.003 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:40:24.002+0000 7fadd68ba640 -1 reset not still connected to 0x55d6d1a04000 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:40:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-18T01:40:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:40:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:40:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:40:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-18T01:40:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:40:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:40:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:40:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:40:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:40:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-18T01:40:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-18T01:40:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-18T01:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:40:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-18T01:40:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-18T01:40:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-09-18T01:40:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:40:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:40:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:40:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-09-18T01:40:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-18T01:40:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:40:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:40:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2024-09-18T01:40:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:40:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:40:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:40:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:40:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:40:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:40:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:40:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-18T01:40:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_28.status 2024-09-18T01:40:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_28.status 2024-09-18T01:40:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-18T01:40:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_28.status 2024-09-18T01:40:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-18T01:40:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-18T01:40:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 140 2024-09-18T01:40:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_28.status 2024-09-18T01:40:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_28.status 2024-09-18T01:40:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-18T01:40:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:40:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=140 2024-09-18T01:40:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:40:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-18T01:40:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=62 2024-09-18T01:40:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 140 = 62 2024-09-18T01:40:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:40:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-18T01:40:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=62 2024-09-18T01:40:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 140 = 62 2024-09-18T01:40:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-18T01:40:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 140 = '' 2024-09-18T01:40:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:40:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.status 2024-09-18T01:40:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-18T01:40:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=140 2024-09-18T01:40:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 140 = 140 2024-09-18T01:40:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-18T01:40:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:40:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-18T01:40:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_28': 2024-09-18T01:40:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:40:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:40:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:40:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b66638d3c5 2024-09-18T01:40:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b66638d3c5 2024-09-18T01:40:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:40:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:40:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:40:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:40:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:44 2024 2024-09-18T01:40:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:44 2024 2024-09-18T01:40:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:44 2024 2024-09-18T01:40:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:40:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:40:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:40:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 7ba84989-9e9f-431c-9a90-7999790fff88 2024-09-18T01:40:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:40:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-18T01:40:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:40:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:40:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_28.export 2024-09-18T01:40:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.export 2024-09-18T01:40:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_28.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.export 2024-09-18T01:40:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_28.export 2024-09-18T01:40:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.export 2024-09-18T01:40:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_28.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.export 2024-09-18T01:40:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_28.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_28.export 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_29 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-18T01:40:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:40:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:40:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:40:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:40:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:40:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-18T01:40:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:40:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:40:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:40:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-18T01:40:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-18T01:40:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-09-18T01:40:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:40:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:40:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:40:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:40:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:40:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:40:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-09-18T01:40:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-18T01:40:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:40:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:40:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:40:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-09-18T01:40:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-18T01:40:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-09-18T01:40:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:40:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:40:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:40:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:40:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:40:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-09-18T01:40:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:40:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:40:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:40:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:40:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:40:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:40:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:40:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-18T01:40:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_29.status 2024-09-18T01:40:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_29.status 2024-09-18T01:40:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-18T01:40:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_29.status 2024-09-18T01:40:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-18T01:40:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-18T01:40:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 142 2024-09-18T01:40:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_29.status 2024-09-18T01:40:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_29.status 2024-09-18T01:40:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-18T01:40:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:40:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=142 2024-09-18T01:40:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:40:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-18T01:40:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=63 2024-09-18T01:40:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 142 = 63 2024-09-18T01:40:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:40:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-18T01:40:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 142 = '' 2024-09-18T01:40:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:40:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-18T01:40:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 142 = '' 2024-09-18T01:40:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:40:35.743 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-18T01:40:35.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-18T01:40:35.744 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-18T01:40:35.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:40:35.743+0000 7f5c70e5d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 49488) UID: 1000 2024-09-18T01:40:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.status 2024-09-18T01:40:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-18T01:40:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=142 2024-09-18T01:40:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 142 = 142 2024-09-18T01:40:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-18T01:40:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:40:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-18T01:40:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_29': 2024-09-18T01:40:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:40:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:40:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:40:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b647b6b3e6 2024-09-18T01:40:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b647b6b3e6 2024-09-18T01:40:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:40:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:50 2024 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:50 2024 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:50 2024 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 044f2e71-cbaa-44f3-9d02-93b1935ce488 2024-09-18T01:40:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:40:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-18T01:40:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:40:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:40:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_29.export 2024-09-18T01:40:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.export 2024-09-18T01:40:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_29.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.export 2024-09-18T01:40:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_29.export 2024-09-18T01:40:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.export 2024-09-18T01:40:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_29.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.export 2024-09-18T01:40:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_29.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_29.export 2024-09-18T01:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_30 2024-09-18T01:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-18T01:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-18T01:40:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:40:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:40:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-18T01:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:40:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:40:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:40:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:40:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-18T01:40:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:40:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:40:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:40:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:40:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:40:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:40:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:40:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:40:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-18T01:40:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-18T01:40:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-18T01:40:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:40:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:40:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:40:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-18T01:40:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-18T01:40:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-09-18T01:40:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:40:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:40:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:40:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-09-18T01:40:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-18T01:40:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2024-09-18T01:40:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:40:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:40:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2024-09-18T01:40:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:40:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:40:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:40:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-18T01:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-18T01:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_30.status 2024-09-18T01:40:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_30.status 2024-09-18T01:40:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-18T01:40:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_30.status 2024-09-18T01:40:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-18T01:40:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-18T01:40:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 144 2024-09-18T01:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_30.status 2024-09-18T01:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_30.status 2024-09-18T01:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-18T01:40:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:40:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=144 2024-09-18T01:40:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:40:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-18T01:40:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=64 2024-09-18T01:40:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 144 = 64 2024-09-18T01:40:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-18T01:40:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:42.745 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-18T01:40:42.745 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:40:42.745 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-18T01:40:42.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-18T01:40:42.746 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-18T01:40:42.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-18T01:40:42.748 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-18T01:40:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 144 = '' 2024-09-18T01:40:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:40:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-18T01:40:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 144 = '' 2024-09-18T01:40:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:40:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.status 2024-09-18T01:40:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-18T01:40:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=144 2024-09-18T01:40:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 144 = 144 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:40:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-18T01:40:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_30': 2024-09-18T01:40:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:40:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b627f85e6d 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b627f85e6d 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:32:56 2024 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:32:56 2024 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:32:56 2024 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:40:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:40:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:40:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 82774ab4-7663-4c2d-b86c-48df151df61b 2024-09-18T01:40:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_30.export 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.export 2024-09-18T01:40:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_30.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.export 2024-09-18T01:40:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_30.export 2024-09-18T01:40:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.export 2024-09-18T01:40:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_30.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.export 2024-09-18T01:40:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_30.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_30.export 2024-09-18T01:40:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:40:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_31 2024-09-18T01:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-18T01:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:40:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-18T01:40:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:40:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:40:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:40:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:40:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:40:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:40:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:40:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:40:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-18T01:40:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-18T01:40:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:40:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:40:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:40:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-09-18T01:40:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-18T01:40:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-09-18T01:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:40:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:40:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-09-18T01:40:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-18T01:40:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2024-09-18T01:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:40:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:40:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:40:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:40:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:40:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:40:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:40:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:40:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:40:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2024-09-18T01:40:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:40:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:40:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:40:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:40:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:40:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:40:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-18T01:40:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-18T01:40:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_31.status 2024-09-18T01:40:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_31.status 2024-09-18T01:40:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-18T01:40:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_31.status 2024-09-18T01:40:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-18T01:40:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-18T01:40:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 146 2024-09-18T01:40:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_31.status 2024-09-18T01:40:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_31.status 2024-09-18T01:40:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-18T01:40:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:40:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=146 2024-09-18T01:40:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:40:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-18T01:40:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=65 2024-09-18T01:40:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 146 = 65 2024-09-18T01:40:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:40:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-18T01:40:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 146 = '' 2024-09-18T01:40:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:40:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-18T01:40:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 146 = '' 2024-09-18T01:40:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:40:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.status 2024-09-18T01:40:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-18T01:40:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=146 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 146 = 146 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:40:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_31': 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6354f5530 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6354f5530 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:40:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:03 2024 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:03 2024 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:03 2024 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 5d13ee44-9f2d-44a2-bfb5-f6f0c940027e 2024-09-18T01:40:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_31.export 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.export 2024-09-18T01:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_31.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.export 2024-09-18T01:40:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_31.export 2024-09-18T01:40:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.export 2024-09-18T01:40:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_31.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.export 2024-09-18T01:40:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_31.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_31.export 2024-09-18T01:40:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:40:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_32 2024-09-18T01:40:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-18T01:40:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:40:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:40:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:40:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-18T01:40:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:40:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:40:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:40:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-18T01:40:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:40:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:40:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:40:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:40:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:40:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:40:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:40:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:40:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:40:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-18T01:40:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:40:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:40:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:40:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:40:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:40:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-18T01:40:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-18T01:40:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-18T01:40:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:40:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:40:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:40:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:40:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:40:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-18T01:40:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-18T01:40:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:40:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:40:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:40:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:40:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2024-09-18T01:40:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-18T01:40:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:40:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:40:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:40:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2024-09-18T01:40:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:40:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:40:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:40:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:40:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:40:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:40:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:40:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:40:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:40:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2024-09-18T01:40:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:40:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:40:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:40:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:40:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:40:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-18T01:40:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_32.status 2024-09-18T01:40:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_32.status 2024-09-18T01:40:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-18T01:40:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_32.status 2024-09-18T01:40:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:40:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:40:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-18T01:40:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-18T01:40:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 148 2024-09-18T01:40:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_32.status 2024-09-18T01:40:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:40:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_32.status 2024-09-18T01:40:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-18T01:40:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:40:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=148 2024-09-18T01:40:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:40:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:40:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:40:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-18T01:40:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=66 2024-09-18T01:40:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 148 = 66 2024-09-18T01:40:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:40:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:40:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:40:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-18T01:40:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 148 = '' 2024-09-18T01:40:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:40:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:40:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:40:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:40:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:40:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:40:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-18T01:40:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:40:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:40:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:40:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 148 = '' 2024-09-18T01:40:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:40:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:41:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:41:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.status 2024-09-18T01:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-18T01:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=148 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 148 = 148 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:41:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-18T01:41:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_32': 2024-09-18T01:41:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:41:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:41:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:41:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6a88ab1ce 2024-09-18T01:41:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6a88ab1ce 2024-09-18T01:41:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:41:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:41:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:41:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:41:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:15 2024 2024-09-18T01:41:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:15 2024 2024-09-18T01:41:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:15 2024 2024-09-18T01:41:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:41:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:41:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:41:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: b485039f-a047-4ba3-86d2-a7ddfa23cdc8 2024-09-18T01:41:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-18T01:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_32.export 2024-09-18T01:41:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.export 2024-09-18T01:41:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_32.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.export 2024-09-18T01:41:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_32.export 2024-09-18T01:41:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.export 2024-09-18T01:41:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_32.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.export 2024-09-18T01:41:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_32.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_32.export 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_33 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-18T01:41:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:41:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:41:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:41:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:41:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:41:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:41:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:41:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:41:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-18T01:41:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-18T01:41:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-18T01:41:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:41:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:41:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:41:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:41:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-18T01:41:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-18T01:41:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-18T01:41:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:41:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:41:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:41:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-18T01:41:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-18T01:41:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-18T01:41:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:41:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:41:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:41:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:41:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:41:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:41:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-18T01:41:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:41:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:41:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:41:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:41:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_33.status 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_33.status 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_33.status 2024-09-18T01:41:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-18T01:41:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-18T01:41:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 150 2024-09-18T01:41:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_33.status 2024-09-18T01:41:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_33.status 2024-09-18T01:41:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-18T01:41:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:41:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=150 2024-09-18T01:41:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:41:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-18T01:41:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=67 2024-09-18T01:41:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 150 = 67 2024-09-18T01:41:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:41:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-18T01:41:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 150 = '' 2024-09-18T01:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:41:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-18T01:41:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 150 = '' 2024-09-18T01:41:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:41:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.status 2024-09-18T01:41:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-18T01:41:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=150 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 150 = 150 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:41:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-18T01:41:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_33': 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6c4d195af 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6c4d195af 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:25 2024 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:25 2024 2024-09-18T01:41:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:25 2024 2024-09-18T01:41:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:41:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:41:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:41:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: a3efecd5-88da-4f50-a556-abf649ac4e9a 2024-09-18T01:41:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:41:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_33.export 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.export 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_33.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.export 2024-09-18T01:41:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_33.export 2024-09-18T01:41:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.export 2024-09-18T01:41:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_33.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.export 2024-09-18T01:41:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_33.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_33.export 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_34 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:41:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-18T01:41:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:41:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:41:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:41:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-18T01:41:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:41:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:41:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-18T01:41:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:41:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:41:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:41:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:41:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:41:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:41:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-18T01:41:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-18T01:41:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:41:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:41:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:41:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:41:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:41:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-09-18T01:41:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-18T01:41:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2024-09-18T01:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:41:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:41:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2024-09-18T01:41:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:41:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:41:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-18T01:41:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_34.status 2024-09-18T01:41:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_34.status 2024-09-18T01:41:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-18T01:41:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_34.status 2024-09-18T01:41:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-18T01:41:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-18T01:41:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 152 2024-09-18T01:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_34.status 2024-09-18T01:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_34.status 2024-09-18T01:41:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-18T01:41:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:41:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=152 2024-09-18T01:41:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:41:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-18T01:41:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=68 2024-09-18T01:41:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 152 = 68 2024-09-18T01:41:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:41:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-18T01:41:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 152 = '' 2024-09-18T01:41:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:41:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-18T01:41:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 152 = '' 2024-09-18T01:41:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:41:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.status 2024-09-18T01:41:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-18T01:41:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=152 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 152 = 152 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:41:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-18T01:41:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_34': 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6cab81a34 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6cab81a34 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:39 2024 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:39 2024 2024-09-18T01:41:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:39 2024 2024-09-18T01:41:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:41:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:41:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:41:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 5d3ed84a-3952-450d-a72a-a0fbf5938579 2024-09-18T01:41:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:41:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-18T01:41:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:41:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:41:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_34.export 2024-09-18T01:41:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.export 2024-09-18T01:41:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_34.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.export 2024-09-18T01:41:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_34.export 2024-09-18T01:41:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.export 2024-09-18T01:41:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_34.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.export 2024-09-18T01:41:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_34.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_34.export 2024-09-18T01:41:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_35 2024-09-18T01:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-18T01:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-18T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-18T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:41:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:41:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:41:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-18T01:41:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:41:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:41:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:41:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:41:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:41:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-18T01:41:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-18T01:41:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:41:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:41:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:41:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-09-18T01:41:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-18T01:41:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2024-09-18T01:41:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:41:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:41:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:41:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:41:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:41:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2024-09-18T01:41:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-18T01:41:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_35 2024-09-18T01:41:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:41:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:41:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:41:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:41:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_35 2024-09-18T01:41:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:41:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:41:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:41:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:41:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:41:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:41:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:41:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_35.status 2024-09-18T01:41:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_35.status 2024-09-18T01:41:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-18T01:41:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_35.status 2024-09-18T01:41:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-18T01:41:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-18T01:41:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 154 2024-09-18T01:41:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_35.status 2024-09-18T01:41:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_35.status 2024-09-18T01:41:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-18T01:41:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:41:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=154 2024-09-18T01:41:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:41:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-18T01:41:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=69 2024-09-18T01:41:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 154 = 69 2024-09-18T01:41:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:41:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-18T01:41:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 154 = '' 2024-09-18T01:41:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:41:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-18T01:41:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 154 = '' 2024-09-18T01:41:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:41:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.status 2024-09-18T01:41:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-18T01:41:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=154 2024-09-18T01:41:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 154 = 154 2024-09-18T01:41:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-18T01:41:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:41:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-18T01:41:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_35': 2024-09-18T01:41:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b663254925 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b663254925 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:52 2024 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:52 2024 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:52 2024 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:41:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:41:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 1650cda4-929a-407a-b124-0d014c9e4fa3 2024-09-18T01:41:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:41:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-18T01:41:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:41:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:41:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_35.export 2024-09-18T01:41:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.export 2024-09-18T01:41:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_35.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.export 2024-09-18T01:41:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_35.export 2024-09-18T01:41:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.export 2024-09-18T01:41:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_35.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.export 2024-09-18T01:41:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_35.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_35.export 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_36 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:41:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-18T01:41:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:41:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:41:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:41:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:41:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:41:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:41:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:41:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:41:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:41:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:41:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:41:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-18T01:41:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-18T01:41:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-18T01:41:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:41:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:41:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:41:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-18T01:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-18T01:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:41:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:41:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:41:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-18T01:41:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-18T01:41:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:41:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:41:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:41:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:41:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-18T01:41:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:41:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:41:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:41:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:41:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:41:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:41:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:41:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-18T01:41:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_36.status 2024-09-18T01:41:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_36.status 2024-09-18T01:41:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-18T01:41:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_36.status 2024-09-18T01:41:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-18T01:41:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-18T01:41:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 156 2024-09-18T01:41:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_36.status 2024-09-18T01:41:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_36.status 2024-09-18T01:41:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-18T01:41:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:41:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=156 2024-09-18T01:41:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:41:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-18T01:41:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=70 2024-09-18T01:41:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 156 = 70 2024-09-18T01:41:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-18T01:41:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 156 = '' 2024-09-18T01:41:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:41:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-18T01:41:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 156 = '' 2024-09-18T01:41:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:41:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.status 2024-09-18T01:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-18T01:41:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=156 2024-09-18T01:41:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 156 = 156 2024-09-18T01:41:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-18T01:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:41:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-18T01:41:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_36': 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b69303a362 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b69303a362 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:33:59 2024 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:33:59 2024 2024-09-18T01:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:33:59 2024 2024-09-18T01:41:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:41:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:41:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:41:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 8455fc5d-6748-424e-ae66-ca58df099087 2024-09-18T01:41:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:41:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-18T01:41:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:41:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:41:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_36.export 2024-09-18T01:41:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.export 2024-09-18T01:41:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_36.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.export 2024-09-18T01:41:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_36.export 2024-09-18T01:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.export 2024-09-18T01:41:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_36.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.export 2024-09-18T01:41:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_36.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_36.export 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_37 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:41:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-18T01:41:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-18T01:41:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:41:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:36.949 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-18T01:41:36.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-18T01:41:36.950 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-18T01:41:36.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:41:36.949+0000 7fa731f1f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 42518) UID: 1000 2024-09-18T01:41:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-18T01:41:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:41:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:41:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:41:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:41:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:41:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-18T01:41:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:41:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:41:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:41:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:41:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-18T01:41:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-18T01:41:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:41:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:41:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:41:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-18T01:41:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-18T01:41:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-09-18T01:41:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:41:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:41:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:41:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:41:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:41:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:41:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:41:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-09-18T01:41:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-18T01:41:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:41:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:41:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:41:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2024-09-18T01:41:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:41:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:41:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:41:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:41:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:41:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-18T01:41:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_37.status 2024-09-18T01:41:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_37.status 2024-09-18T01:41:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-18T01:41:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_37.status 2024-09-18T01:41:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-18T01:41:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-18T01:41:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 158 2024-09-18T01:41:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_37.status 2024-09-18T01:41:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_37.status 2024-09-18T01:41:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-18T01:41:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:41:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=158 2024-09-18T01:41:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:41:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-18T01:41:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=71 2024-09-18T01:41:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 158 = 71 2024-09-18T01:41:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:41:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-18T01:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 158 = '' 2024-09-18T01:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:41:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-18T01:41:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 158 = '' 2024-09-18T01:41:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:41:40.951 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-18T01:41:40.951 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:41:40.951 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-18T01:41:40.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-18T01:41:40.952 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-18T01:41:40.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-18T01:41:40.955 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-09-18T01:41:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.status 2024-09-18T01:41:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-18T01:41:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=158 2024-09-18T01:41:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 158 = 158 2024-09-18T01:41:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-18T01:41:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:41:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-18T01:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_37': 2024-09-18T01:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b69bc27502 2024-09-18T01:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b69bc27502 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:08 2024 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:08 2024 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:08 2024 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 45c73ce1-e604-4921-a91c-891688be3288 2024-09-18T01:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_37.export 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.export 2024-09-18T01:41:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_37.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.export 2024-09-18T01:41:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_37.export 2024-09-18T01:41:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.export 2024-09-18T01:41:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_37.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.export 2024-09-18T01:41:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_37.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_37.export 2024-09-18T01:41:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:41:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_38 2024-09-18T01:41:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-18T01:41:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:41:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-18T01:41:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:41:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:41:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:41:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-18T01:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:41:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:41:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:41:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:41:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-18T01:41:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-18T01:41:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-18T01:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:41:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-18T01:41:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-18T01:41:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2024-09-18T01:41:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:41:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:41:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:41:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:41:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:41:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2024-09-18T01:41:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-18T01:41:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2024-09-18T01:41:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:45.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:45.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:45.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:41:45.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:41:45.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:41:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2024-09-18T01:41:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_38.status 2024-09-18T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_38.status 2024-09-18T01:41:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-18T01:41:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_38.status 2024-09-18T01:41:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-18T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-18T01:41:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 160 2024-09-18T01:41:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_38.status 2024-09-18T01:41:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_38.status 2024-09-18T01:41:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-18T01:41:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:41:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=160 2024-09-18T01:41:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:41:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-18T01:41:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=72 2024-09-18T01:41:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 160 = 72 2024-09-18T01:41:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:41:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-18T01:41:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 160 = '' 2024-09-18T01:41:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-18T01:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 160 = '' 2024-09-18T01:41:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:41:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.status 2024-09-18T01:41:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-18T01:41:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=160 2024-09-18T01:41:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 160 = 160 2024-09-18T01:41:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-18T01:41:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:41:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_38': 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b63d5b8f4a 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b63d5b8f4a 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:41:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:41:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:14 2024 2024-09-18T01:41:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:14 2024 2024-09-18T01:41:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:14 2024 2024-09-18T01:41:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:41:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:41:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:41:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: db636ff1-46c8-441b-8a47-bca8ce4f7f7e 2024-09-18T01:41:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_38.export 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.export 2024-09-18T01:41:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_38.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.export 2024-09-18T01:41:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_38.export 2024-09-18T01:41:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.export 2024-09-18T01:41:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_38.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.export 2024-09-18T01:41:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_38.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_38.export 2024-09-18T01:41:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:41:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_39 2024-09-18T01:41:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-18T01:41:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:41:51.749 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9639226789182187) > weight (0.25) 2024-09-18T01:41:51.749 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-18T01:41:51.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-18T01:41:51.749 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-18T01:41:51.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:41:51.749+0000 7f3429cf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 47032) UID: 1000 2024-09-18T01:41:52.750 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-18T01:41:52.750 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:41:52.750 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-18T01:41:52.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-18T01:41:52.751 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-18T01:41:52.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-18T01:41:52.754 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-09-18T01:41:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-18T01:41:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-18T01:41:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:41:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:41:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:41:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-18T01:41:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:41:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:41:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:41:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:41:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:41:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:41:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:41:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:41:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:41:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:41:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:41:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-18T01:41:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-18T01:41:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:41:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:41:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:41:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:41:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:41:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:41:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-18T01:41:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-18T01:41:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-18T01:41:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:41:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:41:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:41:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:41:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:41:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-18T01:41:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-18T01:41:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:41:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2024-09-18T01:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:41:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:41:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:41:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:41:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2024-09-18T01:41:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:41:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:41:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:41:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:41:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:41:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:41:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:41:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-18T01:41:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_39.status 2024-09-18T01:41:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_39.status 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_39.status 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-18T01:41:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 162 2024-09-18T01:41:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_39.status 2024-09-18T01:41:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:41:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_39.status 2024-09-18T01:41:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-18T01:41:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:41:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=162 2024-09-18T01:41:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:41:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-18T01:41:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=73 2024-09-18T01:41:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 162 = 73 2024-09-18T01:41:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:41:56.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:56.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:56.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:56.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:56.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-18T01:41:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 162 = '' 2024-09-18T01:41:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:41:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-18T01:41:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:41:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:41:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:41:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:41:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:41:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 162 = '' 2024-09-18T01:41:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:41:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:41:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.status 2024-09-18T01:41:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-18T01:41:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:41:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=162 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 162 = 162 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:41:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:41:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:41:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_39': 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b63ee8dce 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b63ee8dce 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:24 2024 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:24 2024 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:24 2024 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 900fab41-f944-4927-8c53-bac7011186e3 2024-09-18T01:41:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:41:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:41:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-18T01:41:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:41:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:41:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:41:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:41:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_39.export 2024-09-18T01:41:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.export 2024-09-18T01:41:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_39.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.export 2024-09-18T01:41:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_39.export 2024-09-18T01:41:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:41:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.export 2024-09-18T01:42:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_39.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.export 2024-09-18T01:42:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_39.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_39.export 2024-09-18T01:42:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_40 2024-09-18T01:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-18T01:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:42:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-18T01:42:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:42:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:42:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:42:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:42:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:42:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:42:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:42:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:42:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:42:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:42:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-18T01:42:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-18T01:42:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-18T01:42:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:42:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:42:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:42:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:42:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-18T01:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-18T01:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:42:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-09-18T01:42:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-18T01:42:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2024-09-18T01:42:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:42:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:42:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:42:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:42:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2024-09-18T01:42:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:42:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:42:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:42:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:42:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:42:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:42:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:42:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-18T01:42:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_40.status 2024-09-18T01:42:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_40.status 2024-09-18T01:42:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-18T01:42:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_40.status 2024-09-18T01:42:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-18T01:42:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-18T01:42:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 164 2024-09-18T01:42:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_40.status 2024-09-18T01:42:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_40.status 2024-09-18T01:42:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-18T01:42:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:42:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=164 2024-09-18T01:42:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-18T01:42:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=74 2024-09-18T01:42:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 164 = 74 2024-09-18T01:42:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:42:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-18T01:42:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 164 = '' 2024-09-18T01:42:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:42:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-18T01:42:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 164 = '' 2024-09-18T01:42:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:42:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.status 2024-09-18T01:42:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-18T01:42:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=164 2024-09-18T01:42:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 164 = 164 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:42:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-18T01:42:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_40': 2024-09-18T01:42:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:42:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:42:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:42:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6963752a 2024-09-18T01:42:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6963752a 2024-09-18T01:42:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:35 2024 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:35 2024 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:35 2024 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: d28cc690-e321-4677-9f77-fba5372ae4cd 2024-09-18T01:42:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:42:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_40.export 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.export 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_40.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.export 2024-09-18T01:42:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_40.export 2024-09-18T01:42:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.export 2024-09-18T01:42:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_40.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.export 2024-09-18T01:42:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_40.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_40.export 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_41 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:42:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-18T01:42:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-18T01:42:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:42:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:42:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:42:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-18T01:42:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:42:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:42:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:42:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:42:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:42:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:42:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:42:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:42:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-18T01:42:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-18T01:42:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:42:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:42:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:42:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:42:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-18T01:42:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-18T01:42:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-18T01:42:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:42:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:42:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:42:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:42:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:42:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-18T01:42:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-18T01:42:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-18T01:42:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:42:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:42:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:42:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-18T01:42:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:42:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:42:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:42:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:42:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:42:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:42:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:42:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-18T01:42:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_41.status 2024-09-18T01:42:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_41.status 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_41.status 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-18T01:42:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 166 2024-09-18T01:42:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_41.status 2024-09-18T01:42:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_41.status 2024-09-18T01:42:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-18T01:42:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:42:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=166 2024-09-18T01:42:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:42:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-18T01:42:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=75 2024-09-18T01:42:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 166 = 75 2024-09-18T01:42:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:42:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-18T01:42:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 166 = '' 2024-09-18T01:42:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:42:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-18T01:42:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 166 = '' 2024-09-18T01:42:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:42:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.status 2024-09-18T01:42:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-18T01:42:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=166 2024-09-18T01:42:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 166 = 166 2024-09-18T01:42:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-18T01:42:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:42:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-18T01:42:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_41': 2024-09-18T01:42:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:42:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:42:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:42:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b68f70055d 2024-09-18T01:42:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b68f70055d 2024-09-18T01:42:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:41 2024 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:41 2024 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:41 2024 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 5be9e68b-589f-432a-ad64-9c1d14313b4c 2024-09-18T01:42:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_41.export 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.export 2024-09-18T01:42:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_41.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.export 2024-09-18T01:42:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_41.export 2024-09-18T01:42:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.export 2024-09-18T01:42:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_41.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.export 2024-09-18T01:42:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_41.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_41.export 2024-09-18T01:42:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:42:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_42 2024-09-18T01:42:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-18T01:42:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:42:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-18T01:42:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-18T01:42:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:42:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-18T01:42:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:42:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:42:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:42:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-18T01:42:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:42:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:42:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:42:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:42:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:42:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:42:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:42:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:42:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-18T01:42:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:42:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:42:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:42:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-18T01:42:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-18T01:42:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-18T01:42:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:42:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:42:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:42:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:42:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-18T01:42:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-18T01:42:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:42:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:42:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:42:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-09-18T01:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-18T01:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_42 2024-09-18T01:42:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:42:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:42:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:42:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:42:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:42:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:42:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_42 2024-09-18T01:42:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:42:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:42:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:42:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:42:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:42:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:42:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:42:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-18T01:42:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_42.status 2024-09-18T01:42:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_42.status 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_42.status 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-18T01:42:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 168 2024-09-18T01:42:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_42.status 2024-09-18T01:42:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_42.status 2024-09-18T01:42:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-18T01:42:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:42:21.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=168 2024-09-18T01:42:21.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:21.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-18T01:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=76 2024-09-18T01:42:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 168 = 76 2024-09-18T01:42:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:42:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-18T01:42:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 168 = '' 2024-09-18T01:42:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-18T01:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 168 = '' 2024-09-18T01:42:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:42:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.status 2024-09-18T01:42:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-18T01:42:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=168 2024-09-18T01:42:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 168 = 168 2024-09-18T01:42:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-18T01:42:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:42:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-18T01:42:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_42': 2024-09-18T01:42:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:42:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:42:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:42:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b689e118a7 2024-09-18T01:42:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b689e118a7 2024-09-18T01:42:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:34:55 2024 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:34:55 2024 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:34:55 2024 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: f8e75016-c79a-43fb-b490-6cd4ba3ee983 2024-09-18T01:42:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:42:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-18T01:42:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:42:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:42:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_42.export 2024-09-18T01:42:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.export 2024-09-18T01:42:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_42.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.export 2024-09-18T01:42:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_42.export 2024-09-18T01:42:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.export 2024-09-18T01:42:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_42.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.export 2024-09-18T01:42:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_42.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_42.export 2024-09-18T01:42:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:42:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_43 2024-09-18T01:42:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-18T01:42:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:42:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-18T01:42:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-18T01:42:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:42:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:42:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:42:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:42:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:42:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:42:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:42:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:42:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:42:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:42:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:42:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:42:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:42:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-18T01:42:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-18T01:42:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:42:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:42:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:42:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:42:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-09-18T01:42:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-18T01:42:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:42:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:42:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:42:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-09-18T01:42:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-18T01:42:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_43 2024-09-18T01:42:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:42:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:42:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:42:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:42:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:42:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:42:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_43 2024-09-18T01:42:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:42:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:42:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:42:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:42:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:42:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:42:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:42:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_43.status 2024-09-18T01:42:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_43.status 2024-09-18T01:42:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-18T01:42:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_43.status 2024-09-18T01:42:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-18T01:42:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-18T01:42:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 170 2024-09-18T01:42:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_43.status 2024-09-18T01:42:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_43.status 2024-09-18T01:42:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-18T01:42:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:42:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=170 2024-09-18T01:42:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:42:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-18T01:42:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=77 2024-09-18T01:42:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 170 = 77 2024-09-18T01:42:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:42:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-18T01:42:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 170 = '' 2024-09-18T01:42:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-18T01:42:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 170 = '' 2024-09-18T01:42:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.status 2024-09-18T01:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-18T01:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=170 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 170 = 170 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:42:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-18T01:42:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_43': 2024-09-18T01:42:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:42:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b62aa4f7d5 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b62aa4f7d5 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:02 2024 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:02 2024 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:02 2024 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:42:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:42:33.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 38e4e214-f1ca-4e80-ac69-0b111c7bccc1 2024-09-18T01:42:33.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:42:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-18T01:42:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:42:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:42:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_43.export 2024-09-18T01:42:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.export 2024-09-18T01:42:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_43.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.export 2024-09-18T01:42:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_43.export 2024-09-18T01:42:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.export 2024-09-18T01:42:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_43.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.export 2024-09-18T01:42:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_43.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_43.export 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_44 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-18T01:42:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:42:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:42:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:42:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-18T01:42:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:42:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:42:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:42:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:42:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:42:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:42:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:42:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:42:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:42:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:42:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-18T01:42:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-18T01:42:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-18T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:42:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:42:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-18T01:42:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-18T01:42:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:42:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:42:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:42:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-09-18T01:42:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-18T01:42:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2024-09-18T01:42:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:42:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:42:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:42:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:42:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:42:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:42:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2024-09-18T01:42:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:42:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:42:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:42:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:42:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:42:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:42:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:42:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-18T01:42:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_44.status 2024-09-18T01:42:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_44.status 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_44.status 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-18T01:42:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 172 2024-09-18T01:42:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_44.status 2024-09-18T01:42:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_44.status 2024-09-18T01:42:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-18T01:42:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:42:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=172 2024-09-18T01:42:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:42:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-18T01:42:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=78 2024-09-18T01:42:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 172 = 78 2024-09-18T01:42:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:42:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-18T01:42:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 172 = '' 2024-09-18T01:42:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:42:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-18T01:42:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 172 = '' 2024-09-18T01:42:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:42:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.status 2024-09-18T01:42:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-18T01:42:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=172 2024-09-18T01:42:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 172 = 172 2024-09-18T01:42:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-18T01:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-18T01:42:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_44': 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6720963cc 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6720963cc 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:11 2024 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:11 2024 2024-09-18T01:42:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:11 2024 2024-09-18T01:42:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:42:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:42:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:42:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: f9dd9870-23fe-40ab-a93a-2441b3c7b9d1 2024-09-18T01:42:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_44.export 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.export 2024-09-18T01:42:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_44.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.export 2024-09-18T01:42:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_44.export 2024-09-18T01:42:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.export 2024-09-18T01:42:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_44.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.export 2024-09-18T01:42:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_44.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_44.export 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_45 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-18T01:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-18T01:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:42:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-18T01:42:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:42:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:42:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:42:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:42:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:42:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-18T01:42:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:42:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:42:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:42:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:42:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:42:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:42:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-18T01:42:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-18T01:42:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:42:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:42:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:42:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-18T01:42:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-18T01:42:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-09-18T01:42:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:42:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:42:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:42:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:42:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-09-18T01:42:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-18T01:42:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2024-09-18T01:42:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:42:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:42:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:42:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:42:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2024-09-18T01:42:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:42:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:42:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:42:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:42:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:42:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:42:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:42:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-18T01:42:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_45.status 2024-09-18T01:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_45.status 2024-09-18T01:42:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-18T01:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_45.status 2024-09-18T01:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-18T01:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-18T01:42:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 174 2024-09-18T01:42:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_45.status 2024-09-18T01:42:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_45.status 2024-09-18T01:42:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-18T01:42:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:42:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=174 2024-09-18T01:42:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:42:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-18T01:42:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=79 2024-09-18T01:42:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 174 = 79 2024-09-18T01:42:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:42:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-18T01:42:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=79 2024-09-18T01:42:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 174 = 79 2024-09-18T01:42:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:42:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-18T01:42:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 174 = '' 2024-09-18T01:42:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:42:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.status 2024-09-18T01:42:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-18T01:42:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=174 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 174 = 174 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:42:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-18T01:42:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_45': 2024-09-18T01:42:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:42:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:42:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:42:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b642e1b2f1 2024-09-18T01:42:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b642e1b2f1 2024-09-18T01:42:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:17 2024 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:17 2024 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:17 2024 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 454fe261-0829-4a00-bce9-9223fa6aeed6 2024-09-18T01:42:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_45.export 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.export 2024-09-18T01:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_45.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.export 2024-09-18T01:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_45.export 2024-09-18T01:42:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.export 2024-09-18T01:42:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:42:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_45.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.export 2024-09-18T01:42:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_45.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_45.export 2024-09-18T01:42:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:42:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_46 2024-09-18T01:42:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-18T01:42:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:42:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:42:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-18T01:42:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-18T01:42:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:42:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:42:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:42:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-18T01:42:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:42:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:42:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:42:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:42:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:42:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:42:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:42:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:42:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:42:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:42:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:42:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-18T01:42:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-18T01:42:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-18T01:42:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:1 2024-09-18T01:42:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-18T01:42:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:1 2024-09-18T01:42:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:1 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=1 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=1 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=1 2024-09-18T01:42:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-18T01:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-18T01:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-18T01:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-18T01:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-18T01:42:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:2 2024-09-18T01:42:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-18T01:42:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:2 2024-09-18T01:42:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=2 2024-09-18T01:42:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=2 2024-09-18T01:42:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-18T01:42:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:2 2024-09-18T01:42:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-18T01:42:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:2 2024-09-18T01:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=2 2024-09-18T01:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-18T01:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=2 2024-09-18T01:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=2 2024-09-18T01:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-18T01:42:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-18T01:42:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-18T01:42:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-18T01:42:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd mirror status : get status for rbd mirror 2024-09-18T01:42:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:admin_socket: invalid command 2024-09-18T01:42:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:3 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:3 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=3 2024-09-18T01:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=3 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:3 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:3 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:42:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=3 2024-09-18T01:42:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-18T01:42:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:42:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:42:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=3 2024-09-18T01:42:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=3 2024-09-18T01:42:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-18T01:42:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-18T01:42:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-09-18T01:42:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:42:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-18T01:42:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_46.status 2024-09-18T01:42:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_46.status 2024-09-18T01:42:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-18T01:42:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_46.status 2024-09-18T01:42:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:42:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-18T01:42:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-18T01:42:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 176 2024-09-18T01:42:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_46.status 2024-09-18T01:42:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:42:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_46.status 2024-09-18T01:42:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-18T01:42:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:42:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=176 2024-09-18T01:42:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:42:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-18T01:42:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=80 2024-09-18T01:42:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 176 = 80 2024-09-18T01:42:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:42:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-18T01:42:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=80 2024-09-18T01:42:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 176 = 80 2024-09-18T01:42:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:42:54.755 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7907092004820127) > weight (0.25) 2024-09-18T01:42:54.755 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6385907678092556) > weight (0.25) 2024-09-18T01:42:54.755 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3980043653875711) > weight (0.25) 2024-09-18T01:42:54.755 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-18T01:42:54.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-18T01:42:54.756 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-18T01:42:54.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:42:54.756+0000 7f253e26b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 36812) UID: 1000 2024-09-18T01:42:54.798 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:42:54.792+0000 7fa22d87d640 -1 reset not still connected to 0x56151f41a000 2024-09-18T01:42:54.798 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:42:54.792+0000 7fa22d87d640 -1 reset not still connected to 0x561521cb3a00 2024-09-18T01:42:54.798 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:42:54.792+0000 7fa22d87d640 -1 reset not still connected to 0x561521cb60d0 2024-09-18T01:42:54.798 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:42:54.792+0000 7fa22d87d640 -1 reset not still connected to 0x5615227dd040 2024-09-18T01:42:54.798 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:42:54.792+0000 7fa22d87d640 -1 reset not still connected to 0x5615252c3860 2024-09-18T01:42:54.798 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:42:54.792+0000 7fa22d87d640 -1 reset not still connected to 0x5615257b9ad0 2024-09-18T01:42:54.798 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:42:54.792+0000 7fa22d87d640 -1 reset not still connected to 0x5615257b9c70 2024-09-18T01:42:54.955 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-18T01:42:54.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-18T01:42:54.956 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-18T01:42:54.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:42:54.956+0000 7f797cc45640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 59059) UID: 1000 2024-09-18T01:42:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-18T01:42:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 176 = '' 2024-09-18T01:42:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:42:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-18T01:42:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 176 = '' 2024-09-18T01:42:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 2 2024-09-18T01:42:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:42:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:42:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:42:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:42:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-18T01:42:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:42:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:42:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:42:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:42:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:42:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:42:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:42:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 176 = '' 2024-09-18T01:42:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:42:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 2 2024-09-18T01:43:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:43:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:43:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:43:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-18T01:43:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:43:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:43:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:43:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:43:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 176 = '' 2024-09-18T01:43:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 4 2024-09-18T01:43:02.757 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-18T01:43:02.757 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:43:02.758 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-18T01:43:02.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-18T01:43:02.758 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-18T01:43:02.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-18T01:43:02.761 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-18T01:43:02.957 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-18T01:43:02.957 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:43:02.957 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-18T01:43:02.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-18T01:43:02.958 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-18T01:43:02.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-18T01:43:02.961 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-18T01:43:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:43:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:43:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.status 2024-09-18T01:43:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-18T01:43:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=176 2024-09-18T01:43:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 176 = 176 2024-09-18T01:43:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-18T01:43:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:43:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:43:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-18T01:43:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_46': 2024-09-18T01:43:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:43:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6f6fe3181 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6f6fe3181 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:27 2024 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:27 2024 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:27 2024 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:43:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:43:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 891bfe82-6886-4816-b990-e7e59c1ed8c5 2024-09-18T01:43:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:43:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-18T01:43:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:43:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:43:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_46.export 2024-09-18T01:43:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.export 2024-09-18T01:43:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_46.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.export 2024-09-18T01:43:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_46.export 2024-09-18T01:43:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.export 2024-09-18T01:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_46.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.export 2024-09-18T01:43:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_46.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_46.export 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_47 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:43:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-18T01:43:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-18T01:43:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:43:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:43:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-18T01:43:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:43:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:43:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:43:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:43:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:43:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:43:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-18T01:43:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:43:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:43:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:43:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:43:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:43:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:43:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:43:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:43:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:43:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:43:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:43:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-18T01:43:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:43:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:43:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:43:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:43:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:43:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:43:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:43:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-18T01:43:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:43:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:43:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:43:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-18T01:43:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:43:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-18T01:43:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_47.status 2024-09-18T01:43:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:43:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_47.status 2024-09-18T01:43:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-18T01:43:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_47.status 2024-09-18T01:43:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:43:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-18T01:43:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-18T01:43:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 178 2024-09-18T01:43:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_47.status 2024-09-18T01:43:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_47.status 2024-09-18T01:43:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-18T01:43:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:43:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=178 2024-09-18T01:43:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:43:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-18T01:43:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=81 2024-09-18T01:43:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 178 = 81 2024-09-18T01:43:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:43:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-18T01:43:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=81 2024-09-18T01:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 178 = 81 2024-09-18T01:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:43:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-18T01:43:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:43:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:43:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 178 = '' 2024-09-18T01:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:43:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.status 2024-09-18T01:43:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-18T01:43:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=178 2024-09-18T01:43:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 178 = 178 2024-09-18T01:43:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-18T01:43:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:43:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-18T01:43:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_47': 2024-09-18T01:43:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:43:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:43:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b62432c3f3 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b62432c3f3 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:39 2024 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:39 2024 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:39 2024 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: c36486c3-b203-4110-86f0-cb987a216048 2024-09-18T01:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:43:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-18T01:43:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:43:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:43:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_47.export 2024-09-18T01:43:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.export 2024-09-18T01:43:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_47.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.export 2024-09-18T01:43:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_47.export 2024-09-18T01:43:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.export 2024-09-18T01:43:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_47.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.export 2024-09-18T01:43:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_47.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_47.export 2024-09-18T01:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_48 2024-09-18T01:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-18T01:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:43:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-18T01:43:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-18T01:43:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:43:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:43:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-18T01:43:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:43:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:43:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:43:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-18T01:43:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:43:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:43:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:43:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:43:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:43:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:43:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:43:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:43:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:43:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:43:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:43:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-18T01:43:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:43:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:43:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:43:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:43:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:43:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:43:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-18T01:43:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:43:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:43:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:43:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:43:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-18T01:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_48.status 2024-09-18T01:43:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:43:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_48.status 2024-09-18T01:43:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-18T01:43:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_48.status 2024-09-18T01:43:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:43:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-18T01:43:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-18T01:43:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 180 2024-09-18T01:43:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_48.status 2024-09-18T01:43:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_48.status 2024-09-18T01:43:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-18T01:43:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:43:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=180 2024-09-18T01:43:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:43:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-18T01:43:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=82 2024-09-18T01:43:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 180 = 82 2024-09-18T01:43:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-18T01:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=82 2024-09-18T01:43:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 180 = 82 2024-09-18T01:43:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:43:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-18T01:43:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:43:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:43:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:43:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:43:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 180 = '' 2024-09-18T01:43:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:43:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.status 2024-09-18T01:43:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-18T01:43:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=180 2024-09-18T01:43:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 180 = 180 2024-09-18T01:43:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-18T01:43:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:43:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-18T01:43:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_48': 2024-09-18T01:43:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:43:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:43:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:43:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b69789c599 2024-09-18T01:43:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b69789c599 2024-09-18T01:43:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:43:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:52 2024 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:52 2024 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:52 2024 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 9bca644c-78a0-405d-b934-3bd6906b0d98 2024-09-18T01:43:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:43:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_48.export 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.export 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_48.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.export 2024-09-18T01:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_48.export 2024-09-18T01:43:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.export 2024-09-18T01:43:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_48.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.export 2024-09-18T01:43:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_48.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_48.export 2024-09-18T01:43:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_49 2024-09-18T01:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-18T01:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:43:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-18T01:43:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:43:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:43:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-18T01:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:43:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-18T01:43:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:43:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:43:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:43:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:43:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:43:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:43:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:43:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_49.status 2024-09-18T01:43:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:43:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_49.status 2024-09-18T01:43:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-18T01:43:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_49.status 2024-09-18T01:43:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:43:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-18T01:43:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-18T01:43:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 182 2024-09-18T01:43:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_49.status 2024-09-18T01:43:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_49.status 2024-09-18T01:43:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-18T01:43:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:43:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=182 2024-09-18T01:43:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:43:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-18T01:43:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=83 2024-09-18T01:43:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 182 = 83 2024-09-18T01:43:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:43:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-18T01:43:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=83 2024-09-18T01:43:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 182 = 83 2024-09-18T01:43:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:43:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-18T01:43:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:43:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:43:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:43:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:43:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 182 = '' 2024-09-18T01:43:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:43:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.status 2024-09-18T01:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-18T01:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=182 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 182 = 182 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:43:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-18T01:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_49': 2024-09-18T01:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:43:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b68718d81c 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b68718d81c 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:35:59 2024 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:35:59 2024 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:35:59 2024 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:43:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:43:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:43:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 2a71d26d-40fd-4228-8efb-d1e4955324d6 2024-09-18T01:43:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:43:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-18T01:43:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:43:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:43:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_49.export 2024-09-18T01:43:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.export 2024-09-18T01:43:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_49.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.export 2024-09-18T01:43:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_49.export 2024-09-18T01:43:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.export 2024-09-18T01:43:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_49.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.export 2024-09-18T01:43:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_49.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_49.export 2024-09-18T01:43:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_50 2024-09-18T01:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-18T01:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-18T01:43:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-18T01:43:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=started 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:43:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-18T01:43:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=started 2024-09-18T01:43:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_result 2024-09-18T01:43:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=stopped 2024-09-18T01:43:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-18T01:43:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -i state 2024-09-18T01:43:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_instance=cluster1 2024-09-18T01:43:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:43:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:43:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance=cluster1 2024-09-18T01:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local loop_instance 2024-09-18T01:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T01:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sleep 0 2024-09-18T01:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-18T01:43:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ seq 0 3 2024-09-18T01:43:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:43:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1:0 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ shift 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local val=cluster1:0 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster_var_name=cluster 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local instance_var_name=instance 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval cluster=cluster1 2024-09-18T01:43:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ eval instance=0 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local local_cluster=cluster1:0 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster=cluster1 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local val=cluster1:0 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local cluster_var_name=local_cluster 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ local instance_var_name=instance 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ cluster=cluster1 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ instance=0 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T01:43:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval local_cluster=cluster1 2024-09-18T01:43:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ local_cluster=cluster1 2024-09-18T01:43:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ eval instance=0 2024-09-18T01:43:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ instance=0 2024-09-18T01:43:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T01:43:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T01:43:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-18T01:43:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ return 0 2024-09-18T01:43:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ status_result=' "state": "Replaying"' 2024-09-18T01:43:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo ' "state": "Replaying"' 2024-09-18T01:43:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ grep -i Replaying 2024-09-18T01:43:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: "state": "Replaying" 2024-09-18T01:43:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=started 2024-09-18T01:43:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test started = started 2024-09-18T01:43:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = journal ']' 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' snapshot = snapshot ']' 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_cluster=cluster1 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-18T01:43:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster2-mirror-image_50.status 2024-09-18T01:43:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local status_log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_50.status 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ echo cluster1-mirror-image_50.status 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local local_status_log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-18T01:43:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Snapshot ID: 184 2024-09-18T01:43:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_50.status 2024-09-18T01:43:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_50.status 2024-09-18T01:43:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-18T01:43:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-18T01:43:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snapshot_id=184 2024-09-18T01:43:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.2 2024-09-18T01:43:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-18T01:43:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=84 2024-09-18T01:43:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 184 = 84 2024-09-18T01:43:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.4 2024-09-18T01:43:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-18T01:43:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=84 2024-09-18T01:43:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 184 = 84 2024-09-18T01:43:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.8 2024-09-18T01:43:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-18T01:43:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:43:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:-:1.1: Document is empty 2024-09-18T01:43:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: 2024-09-18T01:43:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:^ 2024-09-18T01:43:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id= 2024-09-18T01:43:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 184 = '' 2024-09-18T01:43:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-18T01:43:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1.6 2024-09-18T01:43:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local log=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.status 2024-09-18T01:43:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-18T01:43:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-18T01:43:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-18T01:43:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local primary_snapshot_id=184 2024-09-18T01:43:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 184 = 184 2024-09-18T01:43:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-18T01:43:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:43:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:43:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-18T01:43:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:43:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-18T01:43:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_50': 2024-09-18T01:43:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T01:43:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:43:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 2 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 10b6642fd617 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.10b6642fd617 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:36:10 2024 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:36:10 2024 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:36:10 2024 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: protected: False 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:43:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 03d12fa6-b2eb-4ce4-bb75-8944661eb0eb 2024-09-18T01:43:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: false 2024-09-18T01:43:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-18T01:43:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:43:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap_name=snap 2024-09-18T01:43:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local ret=0 2024-09-18T01:43:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local rmt_export=/tmp/tmp.YxZgdY10x3/cluster2-mirror-image_50.export 2024-09-18T01:43:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local loc_export=/tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.export 2024-09-18T01:43:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_50.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.export 2024-09-18T01:43:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_50.export 2024-09-18T01:43:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.export 2024-09-18T01:43:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-18T01:43:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cmp /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_50.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.export 2024-09-18T01:43:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -f /tmp/tmp.YxZgdY10x3/cluster2-mirror-image_50.export /tmp/tmp.YxZgdY10x3/cluster1-mirror-image_50.export 2024-09-18T01:43:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ testlog 'TEST: delete many images' 2024-09-18T01:43:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ tee -a /tmp/tmp.YxZgdY10x3/rbd-mirror.test.log 2024-09-18T01:43:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ date '+%F %T' 2024-09-18T01:43:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo 2024-09-18 01:43:38 TEST: delete many images 2024-09-18T01:43:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:2024-09-18 01:43:38 TEST: delete many images 2024-09-18T01:43:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 50 2024-09-18T01:43:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_1 2024-09-18T01:43:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-18T01:43:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:43:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:43:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-18T01:43:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:43:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-18T01:43:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:43:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:43:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:43:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_1 2024-09-18T01:43:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:43:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-18T01:43:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-18T01:43:39.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:39.158+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaf040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:43:39.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:39.158+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f39bb80 [3/e824ac62-3a6a-4a7a-9f9f-f4f31117d463] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:43:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:43:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_2 2024-09-18T01:43:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-18T01:43:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:43:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:43:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-18T01:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:43:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-18T01:43:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:43:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:43:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:43:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_2 2024-09-18T01:43:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:43:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-18T01:43:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-18T01:43:42.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:42.201+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaa680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:43:42.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:42.201+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f39b180 [3/9dc75a63-a2d8-4ff1-bf1e-f7d6b7f28a2c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:43:42.821 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:43:42.819+0000 7fac12995640 -1 reset not still connected to 0x55e4eb2e5ba0 2024-09-18T01:43:42.821 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:43:42.819+0000 7fac12995640 -1 reset not still connected to 0x55e4eb8500d0 2024-09-18T01:43:42.821 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:43:42.819+0000 7fac12995640 -1 reset not still connected to 0x55e4eb850340 2024-09-18T01:43:42.821 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:43:42.819+0000 7fac12995640 -1 reset not still connected to 0x55e4ec5968f0 2024-09-18T01:43:42.821 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:43:42.819+0000 7fac12995640 -1 reset not still connected to 0x55e4ec596c30 2024-09-18T01:43:42.821 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:43:42.819+0000 7fac12995640 -1 reset not still connected to 0x55e4ec596ea0 2024-09-18T01:43:42.822 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:43:42.819+0000 7fac12995640 -1 reset not still connected to 0x55e4ec597ba0 2024-09-18T01:43:42.822 INFO:tasks.ceph.cluster1.osd.2.smithi066.stderr:2024-09-18T01:43:42.819+0000 7fac12995640 -1 reset not still connected to 0x55e4ec9a9ba0 2024-09-18T01:43:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_3 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:43:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-18T01:43:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:43:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-18T01:43:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:43:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:43:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:43:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_3 2024-09-18T01:43:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:43:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-18T01:43:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-18T01:43:45.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:45.263+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaeb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:43:45.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:45.263+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f39a780 [3/4cad14e9-5f0f-48cb-acf6-96241971c0cd] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:43:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_4 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:43:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-18T01:43:47.886 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.883+0000 7fadd68ba640 -1 reset not still connected to 0x55d6b408e4e0 2024-09-18T01:43:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.888+0000 7fadd68ba640 -1 reset not still connected to 0x55d6b408e680 2024-09-18T01:43:47.893 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.890+0000 7fadd68ba640 -1 reset not still connected to 0x55d6bc594820 2024-09-18T01:43:47.896 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.895+0000 7fadd68ba640 -1 reset not still connected to 0x55d6bc5948f0 2024-09-18T01:43:47.896 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.895+0000 7fadd68ba640 -1 reset not still connected to 0x55d6bf5b8270 2024-09-18T01:43:47.896 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.895+0000 7fadd68ba640 -1 reset not still connected to 0x55d6c3d3a4e0 2024-09-18T01:43:47.896 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.895+0000 7fadd68ba640 -1 reset not still connected to 0x55d6c5dbfba0 2024-09-18T01:43:47.897 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.896+0000 7fadd68ba640 -1 reset not still connected to 0x55d6cb719ad0 2024-09-18T01:43:47.897 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.896+0000 7fadd68ba640 -1 reset not still connected to 0x55d6cb71a270 2024-09-18T01:43:47.897 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.896+0000 7fadd68ba640 -1 reset not still connected to 0x55d6cb71a680 2024-09-18T01:43:47.897 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.896+0000 7fadd68ba640 -1 reset not still connected to 0x55d6cb71a8f0 2024-09-18T01:43:47.897 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.896+0000 7fadd68ba640 -1 reset not still connected to 0x55d6cb99a8f0 2024-09-18T01:43:47.897 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.896+0000 7fadd68ba640 -1 reset not still connected to 0x55d6d116c270 2024-09-18T01:43:47.898 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:43:47.896+0000 7fadd68ba640 -1 reset not still connected to 0x55d6d19521a0 2024-09-18T01:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:43:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-18T01:43:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:43:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:43:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:43:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_4 2024-09-18T01:43:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:43:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-18T01:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-18T01:43:48.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:48.219+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaf1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:43:48.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:48.219+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f39ac80 [3/69e2ede3-6b24-471e-8e85-a7ce3370267a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:43:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:43:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_5 2024-09-18T01:43:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-18T01:43:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:43:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:43:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-18T01:43:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:43:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-18T01:43:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:43:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:43:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_5 2024-09-18T01:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:43:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-18T01:43:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-18T01:43:50.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:50.308+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecb44e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:43:50.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:50.308+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f39b900 [3/f65227d5-766c-440e-beb8-127249467502] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:43:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:43:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_6 2024-09-18T01:43:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-18T01:43:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:43:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:43:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-18T01:43:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:43:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-18T01:43:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:43:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:43:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:43:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_6 2024-09-18T01:43:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:43:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-18T01:43:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-18T01:43:53.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:53.313+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecb4680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:43:53.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:53.313+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f39af00 [3/72ab9ec5-f35c-428f-a831-18f79d59a14f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:43:54.905 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:43:54.902+0000 7fa22d87d640 -1 reset not still connected to 0x56152596d1e0 2024-09-18T01:43:54.905 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:43:54.902+0000 7fa22d87d640 -1 reset not still connected to 0x56152596d930 2024-09-18T01:43:54.907 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:43:54.906+0000 7fa22d87d640 -1 reset not still connected to 0x5615280f9d40 2024-09-18T01:43:54.907 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:43:54.906+0000 7fa22d87d640 -1 reset not still connected to 0x561528fb5930 2024-09-18T01:43:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_7 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:43:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-18T01:43:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:43:56.178 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:43:56.177+0000 7f678fb95640 -1 reset not still connected to 0x56466d796c30 2024-09-18T01:43:56.178 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:43:56.177+0000 7f678fb95640 -1 reset not still connected to 0x564674402750 2024-09-18T01:43:56.178 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:43:56.177+0000 7f678fb95640 -1 reset not still connected to 0x5646762a2f70 2024-09-18T01:43:56.179 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:43:56.178+0000 7f678fb95640 -1 reset not still connected to 0x564680c820d0 2024-09-18T01:43:56.179 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:43:56.178+0000 7f678fb95640 -1 reset not still connected to 0x56468195cea0 2024-09-18T01:43:56.179 INFO:tasks.ceph.cluster2.osd.1.smithi115.stderr:2024-09-18T01:43:56.178+0000 7f678fb95640 -1 reset not still connected to 0x56468195d040 2024-09-18T01:43:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-18T01:43:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:43:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:43:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:43:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_7 2024-09-18T01:43:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:43:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-18T01:43:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-18T01:43:56.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:56.337+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecb04e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:43:56.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:56.337+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083e60a280 [3/4597c48a-8662-495e-a897-2ee7f1408d91] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:43:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:43:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:43:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:43:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_8 2024-09-18T01:43:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-18T01:43:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:43:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:43:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-18T01:43:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:43:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-18T01:43:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:43:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:43:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:43:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_8 2024-09-18T01:43:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:43:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:43:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:43:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-18T01:43:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-18T01:43:59.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:59.311+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecb0680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:43:59.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:43:59.311+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f39b400 [3/b4e5b9d2-efbb-4d6e-b309-ef7dc24ba029] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_9 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-18T01:44:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-18T01:44:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:44:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_9 2024-09-18T01:44:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:44:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-18T01:44:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-18T01:44:02.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:02.360+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecb0820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:02.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:02.360+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f39aa00 [3/68d70003-8fef-4a72-b907-b819e314282e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_10 2024-09-18T01:44:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-18T01:44:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:44:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-18T01:44:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-18T01:44:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:44:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_10 2024-09-18T01:44:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:44:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-18T01:44:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-18T01:44:05.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:05.376+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec8a9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:05.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:05.376+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f39b680 [3/c330e23e-03a4-45ae-9fa0-73ca0a1ac911] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_11 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-18T01:44:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-18T01:44:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:44:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_11 2024-09-18T01:44:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:44:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-18T01:44:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-18T01:44:08.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:08.384+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec8ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:08.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:08.384+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f3cea00 [3/f4d6f564-c302-4459-b459-852582eb1804] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_12 2024-09-18T01:44:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-18T01:44:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:44:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-18T01:44:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-18T01:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_12 2024-09-18T01:44:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:44:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-18T01:44:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-18T01:44:11.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:11.390+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ea0e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:11.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:11.390+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f3cf180 [3/d652dfd4-0db3-409b-a60f-853dfb4b8bbf] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_13 2024-09-18T01:44:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-18T01:44:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:44:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-18T01:44:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-18T01:44:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:44:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_13 2024-09-18T01:44:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:44:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-18T01:44:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-18T01:44:14.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:14.319+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec8ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:14.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:14.319+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f3cf400 [3/0b20a1f2-779a-42c1-bde9-cc45daa1eb9b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_14 2024-09-18T01:44:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-18T01:44:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:44:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-18T01:44:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-18T01:44:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:44:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_14 2024-09-18T01:44:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:44:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-18T01:44:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-18T01:44:17.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:17.432+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eab24e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:17.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:17.432+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f3cec80 [3/3c45b3f0-662d-4182-8d89-837d0b48bc1a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_15 2024-09-18T01:44:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-18T01:44:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:44:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-18T01:44:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-18T01:44:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:44:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_15 2024-09-18T01:44:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:44:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-18T01:44:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-18T01:44:20.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:20.433+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecb09c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:20.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:20.433+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f3cef00 [3/15b04f61-470a-4fa7-811f-b4270495f8f7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:20.819 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x561522bcedd0 2024-09-18T01:44:20.819 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x56152596cd00 2024-09-18T01:44:20.819 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x56152596d1e0 2024-09-18T01:44:20.819 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x56152596d2b0 2024-09-18T01:44:20.819 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x56152596d380 2024-09-18T01:44:20.820 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x56152596d450 2024-09-18T01:44:20.820 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x56152596d5f0 2024-09-18T01:44:20.820 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x56152596d930 2024-09-18T01:44:20.820 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x5615259c3930 2024-09-18T01:44:20.820 INFO:tasks.ceph.cluster1.osd.0.smithi066.stderr:2024-09-18T01:44:20.819+0000 7fa22d87d640 -1 reset not still connected to 0x561528fb5930 2024-09-18T01:44:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_16 2024-09-18T01:44:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-18T01:44:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:44:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-18T01:44:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-18T01:44:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:44:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_16 2024-09-18T01:44:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:44:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-18T01:44:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-18T01:44:23.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:23.442+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eab29c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:23.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:23.442+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f3cf680 [3/0a9c0c62-9c63-4428-abf9-46d978cda72b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_17 2024-09-18T01:44:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-18T01:44:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:44:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-18T01:44:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-18T01:44:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:44:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_17 2024-09-18T01:44:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:44:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-18T01:44:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-18T01:44:26.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:26.328+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecb01a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:26.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:26.328+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f164280 [3/2af45ec2-1273-4f89-9d4a-659567f6c7cc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_18 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-18T01:44:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-18T01:44:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:44:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_18 2024-09-18T01:44:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:44:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-18T01:44:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-18T01:44:29.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:29.327+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec9e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:29.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:29.327+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f3cf900 [3/e28fa577-03c0-4443-8c02-6edcbe8d5994] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_19 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-18T01:44:31.961 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2928968934234496) > weight (0.25) 2024-09-18T01:44:31.961 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7806821944403902) > weight (0.25) 2024-09-18T01:44:31.961 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7698061348685321) > weight (0.25) 2024-09-18T01:44:31.962 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.2638037928534531) > weight (0.25) 2024-09-18T01:44:31.962 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-18T01:44:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-18T01:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_19 2024-09-18T01:44:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-18T01:44:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-18T01:44:32.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:32.346+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eec3ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:32.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:32.346+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f164500 [3/fc033df2-2782-4b60-b444-9993613dbf5d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_20 2024-09-18T01:44:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-18T01:44:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:44:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-18T01:44:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:35.191 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:44:35.190+0000 7f78fa24b640 -1 reset not still connected to 0x55b9d8ebcb60 2024-09-18T01:44:35.192 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:44:35.190+0000 7f78fa24b640 -1 reset not still connected to 0x55b9df552c30 2024-09-18T01:44:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-18T01:44:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:44:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_20 2024-09-18T01:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-18T01:44:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-18T01:44:35.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:35.368+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec8a340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:35.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:35.368+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f164000 [3/b7c93f86-7ae8-4602-a345-dc61cce15b17] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_21 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-18T01:44:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-18T01:44:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:44:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_21 2024-09-18T01:44:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:44:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-18T01:44:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-18T01:44:38.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:38.572+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec8a4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:38.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:38.572+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f165b80 [3/854098b9-42e8-43d8-b6c2-a45666ee66c5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_22 2024-09-18T01:44:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-18T01:44:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:44:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-18T01:44:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-18T01:44:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:44:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_22 2024-09-18T01:44:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:44:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-18T01:44:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-18T01:44:41.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:41.566+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec8a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:41.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:41.566+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f3cfb80 [3/b41be30f-1770-49fc-9175-aad90b2449de] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_23 2024-09-18T01:44:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-18T01:44:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:44:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-18T01:44:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:44.156 INFO:tasks.ceph.cluster2.osd.0.smithi115.stderr:2024-09-18T01:44:44.154+0000 7fadd68ba640 -1 reset not still connected to 0x55d6d116c270 2024-09-18T01:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-18T01:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_23 2024-09-18T01:44:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:44:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-18T01:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-18T01:44:44.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:44.339+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec8a820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:44.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:44.339+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f165400 [3/0b0b9342-5291-4bb5-ade8-4fa8d9ff735c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_24 2024-09-18T01:44:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-18T01:44:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:44:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-18T01:44:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-18T01:44:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:44:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_24 2024-09-18T01:44:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:44:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-18T01:44:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-18T01:44:47.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:47.394+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec98820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:47.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:47.394+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f670000 [3/6425ebaf-56a7-47a1-9547-1c0136cbba11] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_25 2024-09-18T01:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-18T01:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-18T01:44:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-18T01:44:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:44:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_25 2024-09-18T01:44:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:44:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-18T01:44:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-18T01:44:50.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:50.405+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecafd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:50.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:50.405+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f165680 [3/0ee8e883-4767-4c92-9030-1bb0af68068c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_26 2024-09-18T01:44:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-18T01:44:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:44:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-18T01:44:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-18T01:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_26 2024-09-18T01:44:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:44:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-18T01:44:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-18T01:44:53.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:53.414+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eab3040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:53.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:53.414+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f670280 [3/41d5e821-71ed-41c3-afd1-9e875de13804] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_27 2024-09-18T01:44:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-18T01:44:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:44:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-18T01:44:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-18T01:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_27 2024-09-18T01:44:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:44:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-18T01:44:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-18T01:44:56.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:56.432+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaf6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:56.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:56.432+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f164a00 [3/ba4cfda3-b609-4e89-9671-ed59bd54d468] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:44:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:44:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:44:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:44:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_28 2024-09-18T01:44:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-18T01:44:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:44:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:44:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-18T01:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:44:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-18T01:44:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:44:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:44:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:44:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_28 2024-09-18T01:44:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:44:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:44:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:44:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-18T01:44:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-18T01:44:59.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:59.327+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eab3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:44:59.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:44:59.327+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f670500 [3/7ba84989-9e9f-431c-9a90-7999790fff88] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:00.762 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4883110504318695) > weight (0.25) 2024-09-18T01:45:00.762 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7118125139485867) > weight (0.25) 2024-09-18T01:45:00.762 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.43508304259624) > weight (0.25) 2024-09-18T01:45:00.762 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4890335434964558) > weight (0.25) 2024-09-18T01:45:00.762 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-09-18T01:45:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_29 2024-09-18T01:45:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-18T01:45:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:45:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-18T01:45:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-18T01:45:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:45:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_29 2024-09-18T01:45:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:45:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-18T01:45:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-18T01:45:02.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:02.446+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaf380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:02.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:02.446+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f670780 [3/044f2e71-cbaa-44f3-9d02-93b1935ce488] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_30 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-18T01:45:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-18T01:45:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:45:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_30 2024-09-18T01:45:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:45:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-18T01:45:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-18T01:45:05.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:05.451+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eec44e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:05.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:05.451+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f164c80 [3/82774ab4-7663-4c2d-b86c-48df151df61b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_31 2024-09-18T01:45:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-18T01:45:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:45:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-18T01:45:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-18T01:45:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:45:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_31 2024-09-18T01:45:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:45:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-18T01:45:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-18T01:45:08.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:08.680+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eebfa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:08.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:08.680+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f670a00 [3/5d13ee44-9f2d-44a2-bfb5-f6f0c940027e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_32 2024-09-18T01:45:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-18T01:45:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:45:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-18T01:45:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-18T01:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_32 2024-09-18T01:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-18T01:45:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-18T01:45:11.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:11.479+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eebf040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:11.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:11.479+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f6b2780 [3/b485039f-a047-4ba3-86d2-a7ddfa23cdc8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_33 2024-09-18T01:45:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-18T01:45:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:45:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-18T01:45:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-18T01:45:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:45:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_33 2024-09-18T01:45:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:45:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-18T01:45:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-18T01:45:14.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:14.336+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eebe9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:14.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:14.336+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f6b2a00 [3/a3efecd5-88da-4f50-a556-abf649ac4e9a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_34 2024-09-18T01:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-18T01:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-18T01:45:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-18T01:45:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:45:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_34 2024-09-18T01:45:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:45:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-18T01:45:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-18T01:45:17.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:17.496+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eab3380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:17.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:17.496+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f164780 [3/5d3ed84a-3952-450d-a72a-a0fbf5938579] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:19.168 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:45:19.167+0000 7f78fa24b640 -1 reset not still connected to 0x55b9e0c93d40 2024-09-18T01:45:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_35 2024-09-18T01:45:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-18T01:45:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:45:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-18T01:45:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-18T01:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_35 2024-09-18T01:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-18T01:45:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-18T01:45:20.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:20.533+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eab36c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:20.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:20.533+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f6b2c80 [3/1650cda4-929a-407a-b124-0d014c9e4fa3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_36 2024-09-18T01:45:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-18T01:45:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:45:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-18T01:45:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-18T01:45:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:45:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_36 2024-09-18T01:45:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:45:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-18T01:45:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-18T01:45:23.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:23.516+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:23.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:23.516+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f6b2f00 [3/8455fc5d-6748-424e-ae66-ca58df099087] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_37 2024-09-18T01:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-18T01:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:45:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-18T01:45:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-18T01:45:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:45:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_37 2024-09-18T01:45:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:45:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-18T01:45:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-18T01:45:26.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:26.355+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec9eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:26.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:26.355+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f164f00 [3/45c73ce1-e604-4921-a91c-891688be3288] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_38 2024-09-18T01:45:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-18T01:45:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:45:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-18T01:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-18T01:45:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:45:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_38 2024-09-18T01:45:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:45:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-18T01:45:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-18T01:45:29.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:29.336+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eebe4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:29.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:29.336+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f6b3180 [3/db636ff1-46c8-441b-8a47-bca8ce4f7f7e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_39 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-18T01:45:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-18T01:45:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:45:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_39 2024-09-18T01:45:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:45:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-18T01:45:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-18T01:45:32.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:32.352+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eebf520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:32.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:32.352+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f6b3900 [3/900fab41-f944-4927-8c53-bac7011186e3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_40 2024-09-18T01:45:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-18T01:45:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:45:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-18T01:45:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-18T01:45:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:45:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_40 2024-09-18T01:45:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:45:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-18T01:45:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-18T01:45:35.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:35.373+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecab1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:35.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:35.373+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f6b3b80 [3/d28cc690-e321-4677-9f77-fba5372ae4cd] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_41 2024-09-18T01:45:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-18T01:45:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:45:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-18T01:45:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-18T01:45:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:45:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_41 2024-09-18T01:45:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:45:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-18T01:45:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-18T01:45:38.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:38.395+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eebe1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:38.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:38.395+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083e9d4000 [3/5be9e68b-589f-432a-ad64-9c1d14313b4c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_42 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-18T01:45:40.963 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5647102466208109) > weight (0.25) 2024-09-18T01:45:40.963 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-18T01:45:40.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-18T01:45:40.964 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-18T01:45:40.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:45:40.964+0000 7efef2e52640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41947) UID: 1000 2024-09-18T01:45:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-18T01:45:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:45:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_42 2024-09-18T01:45:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:45:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-18T01:45:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-18T01:45:41.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:41.416+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eebeb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:41.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:41.416+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f6b3400 [3/f8e75016-c79a-43fb-b490-6cd4ba3ee983] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:41.900 INFO:tasks.ceph.cluster1.osd.1.smithi066.stderr:2024-09-18T01:45:41.898+0000 7f5845f5b640 -1 reset not still connected to 0x5575a3038270 2024-09-18T01:45:41.900 INFO:tasks.ceph.cluster1.osd.1.smithi066.stderr:2024-09-18T01:45:41.898+0000 7f5845f5b640 -1 reset not still connected to 0x5575a46c8410 2024-09-18T01:45:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_43 2024-09-18T01:45:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-18T01:45:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:45:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-18T01:45:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-18T01:45:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:45:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_43 2024-09-18T01:45:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:45:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-18T01:45:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-18T01:45:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:44.346+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ee96340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:44.346+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f671900 [3/38e4e214-f1ca-4e80-ac69-0b111c7bccc1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_44 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-18T01:45:46.964 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-18T01:45:46.964 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:45:46.965 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-18T01:45:46.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-18T01:45:46.965 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-18T01:45:46.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-18T01:45:46.968 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-18T01:45:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-18T01:45:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:45:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_44 2024-09-18T01:45:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:45:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-18T01:45:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-18T01:45:47.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:47.416+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec98ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:47.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:47.416+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083e9d4280 [3/f9dd9870-23fe-40ab-a93a-2441b3c7b9d1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_45 2024-09-18T01:45:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-18T01:45:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:45:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-18T01:45:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-18T01:45:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:45:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_45 2024-09-18T01:45:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:45:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-18T01:45:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-18T01:45:50.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:50.425+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec9e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:50.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:50.425+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083e9d4500 [3/454fe261-0829-4a00-bce9-9223fa6aeed6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_46 2024-09-18T01:45:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-18T01:45:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:45:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-18T01:45:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-18T01:45:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:45:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_46 2024-09-18T01:45:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:45:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-18T01:45:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-18T01:45:53.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:53.443+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083edf6340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:53.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:53.443+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083e9d4780 [3/891bfe82-6886-4816-b990-e7e59c1ed8c5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_47 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-18T01:45:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-18T01:45:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:45:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_47 2024-09-18T01:45:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:45:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-18T01:45:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-18T01:45:56.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:56.446+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083edf6000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:56.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:56.446+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083e9d4a00 [3/c36486c3-b203-4110-86f0-cb987a216048] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:45:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:45:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:45:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:45:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_48 2024-09-18T01:45:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-18T01:45:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:45:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:45:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-18T01:45:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:45:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-18T01:45:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:45:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:45:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:45:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_48 2024-09-18T01:45:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:45:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:45:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:45:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-18T01:45:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-18T01:45:59.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:59.339+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec98d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:45:59.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:45:59.339+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083e9d4c80 [3/9bca644c-78a0-405d-b934-3bd6906b0d98] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:46:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:46:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_49 2024-09-18T01:46:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-18T01:46:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:46:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:46:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-18T01:46:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:46:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-18T01:46:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:46:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:46:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:46:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_49 2024-09-18T01:46:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-18T01:46:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-18T01:46:02.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:46:02.476+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eec61a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:46:02.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:46:02.476+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f671b80 [3/2a71d26d-40fd-4228-8efb-d1e4955324d6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:46:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:46:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_50 2024-09-18T01:46:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-18T01:46:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:46:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local snap=snap 2024-09-18T01:46:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-18T01:46:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing snap: 100% complete...done. 2024-09-18T01:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-18T01:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:46:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_50 2024-09-18T01:46:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:46:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-18T01:46:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-18T01:46:05.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:46:05.490+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:46:05.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:46:05.490+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f670c80 [3/03d12fa6-b2eb-4ce4-bb75-8944661eb0eb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:46:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-18T01:46:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-18T01:46:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ tee -a /tmp/tmp.YxZgdY10x3/rbd-mirror.test.log 2024-09-18T01:46:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ date '+%F %T' 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo 2024-09-18 01:46:07 TEST: image deletions should propagate 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:2024-09-18 01:46:07 TEST: image deletions should propagate 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_count=0 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local count 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local last_count=0 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ true 2024-09-18T01:46:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 40 2024-09-18T01:46:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in `seq 1 40` 2024-09-18T01:46:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 1 -ne 1 2024-09-18T01:46:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-18T01:46:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep 'images: ' 2024-09-18T01:46:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cut '-d ' -f 2 2024-09-18T01:46:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: failed to query services: (13) Permission denied 2024-09-18T01:46:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ count=0 2024-09-18T01:46:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 = 0 2024-09-18T01:46:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-18T01:46:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state 2024-09-18T01:46:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 40 2024-09-18T01:46:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in `seq 1 40` 2024-09-18T01:46:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 1 -ne 1 2024-09-18T01:46:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-18T01:46:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep 'image health:' 2024-09-18T01:46:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cut '-d ' -f 3 2024-09-18T01:46:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: failed to query services: (13) Permission denied 2024-09-18T01:46:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ state=OK 2024-09-18T01:46:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test OK = ERROR 2024-09-18T01:46:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test OK = OK 2024-09-18T01:46:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 50 2024-09-18T01:46:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_1 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_1 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_1 2024-09-18T01:46:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-18T01:46:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-18T01:46:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-18T01:46:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_1 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_1 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-18T01:46:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-18T01:46:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_2 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_2 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_2 2024-09-18T01:46:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-18T01:46:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-18T01:46:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-18T01:46:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:46:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_2 2024-09-18T01:46:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_2 2024-09-18T01:46:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-18T01:46:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_3 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_3 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_3 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-18T01:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-18T01:46:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_3 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_3 2024-09-18T01:46:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-18T01:46:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-18T01:46:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_4 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_4 2024-09-18T01:46:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_4 2024-09-18T01:46:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-18T01:46:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-18T01:46:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-18T01:46:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_4 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_4 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-18T01:46:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-18T01:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_5 2024-09-18T01:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-18T01:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_5 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_5 2024-09-18T01:46:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-18T01:46:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-18T01:46:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_5 2024-09-18T01:46:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_5 2024-09-18T01:46:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-18T01:46:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_6 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_6 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_6 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-18T01:46:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-18T01:46:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_6 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_6 2024-09-18T01:46:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-18T01:46:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-18T01:46:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_7 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_7 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_7 2024-09-18T01:46:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-18T01:46:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-18T01:46:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_7 2024-09-18T01:46:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_7 2024-09-18T01:46:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-18T01:46:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-18T01:46:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_8 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:46:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_8 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_8 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-18T01:46:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-18T01:46:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_8 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_8 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-18T01:46:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-18T01:46:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_9 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_9 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_9 2024-09-18T01:46:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-18T01:46:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-18T01:46:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_9 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_9 2024-09-18T01:46:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-18T01:46:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_10 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:46:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_10 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_10 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-18T01:46:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-18T01:46:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-18T01:46:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:46:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_10 2024-09-18T01:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_10 2024-09-18T01:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-18T01:46:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-18T01:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_11 2024-09-18T01:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-18T01:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_11 2024-09-18T01:46:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_11 2024-09-18T01:46:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-18T01:46:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-18T01:46:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-18T01:46:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_11 2024-09-18T01:46:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_11 2024-09-18T01:46:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_11 2024-09-18T01:46:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-18T01:46:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-18T01:46:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_12 2024-09-18T01:46:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_12 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_12 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-18T01:46:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-18T01:46:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_12 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_12 2024-09-18T01:46:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_12 2024-09-18T01:46:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-18T01:46:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_13 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:46:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_13 2024-09-18T01:46:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_13 2024-09-18T01:46:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-18T01:46:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-18T01:46:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_13 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_13 2024-09-18T01:46:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_13 2024-09-18T01:46:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-18T01:46:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-18T01:46:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_14 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_14 2024-09-18T01:46:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_14 2024-09-18T01:46:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-18T01:46:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-18T01:46:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-18T01:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_14 2024-09-18T01:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_14 2024-09-18T01:46:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_14 2024-09-18T01:46:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-18T01:46:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-18T01:46:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_15 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_15 2024-09-18T01:46:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_15 2024-09-18T01:46:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-18T01:46:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-18T01:46:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_15 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_15 2024-09-18T01:46:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_15 2024-09-18T01:46:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-18T01:46:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-18T01:46:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_16 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_16 2024-09-18T01:46:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_16 2024-09-18T01:46:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-18T01:46:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-18T01:46:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-18T01:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_16 2024-09-18T01:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_16 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_16 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-18T01:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-18T01:46:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_17 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_17 2024-09-18T01:46:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_17 2024-09-18T01:46:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-18T01:46:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-18T01:46:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-18T01:46:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_17 2024-09-18T01:46:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_17 2024-09-18T01:46:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_17 2024-09-18T01:46:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-18T01:46:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-18T01:46:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_18 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_18 2024-09-18T01:46:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_18 2024-09-18T01:46:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-18T01:46:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-18T01:46:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_18 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_18 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_18 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-18T01:46:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-18T01:46:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_19 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_19 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_19 2024-09-18T01:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-18T01:46:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-18T01:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_19 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_19 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_19 2024-09-18T01:46:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-18T01:46:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_20 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:46:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_20 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_20 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-18T01:46:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-18T01:46:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-18T01:46:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_20 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_20 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_20 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-18T01:46:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-18T01:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_21 2024-09-18T01:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-18T01:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_21 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_21 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-18T01:46:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-18T01:46:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_21 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_21 2024-09-18T01:46:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_21 2024-09-18T01:46:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-18T01:46:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-18T01:46:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_22 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_22 2024-09-18T01:46:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_22 2024-09-18T01:46:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-18T01:46:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-18T01:46:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-18T01:46:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_22 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_22 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_22 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-18T01:46:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_23 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:46:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_23 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_23 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-18T01:46:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-18T01:46:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_23 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_23 2024-09-18T01:46:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_23 2024-09-18T01:46:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-18T01:46:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-18T01:46:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_24 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_24 2024-09-18T01:46:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_24 2024-09-18T01:46:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-18T01:46:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-18T01:46:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-18T01:46:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_24 2024-09-18T01:46:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_24 2024-09-18T01:46:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_24 2024-09-18T01:46:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-18T01:46:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_25 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_25 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_25 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-18T01:46:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-18T01:46:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-18T01:46:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_25 2024-09-18T01:46:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_25 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_25 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-18T01:46:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_26 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_26 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_26 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-18T01:46:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-18T01:46:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_26 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_26 2024-09-18T01:46:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_26 2024-09-18T01:46:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-18T01:46:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-18T01:46:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_27 2024-09-18T01:46:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_27 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_27 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-18T01:46:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-18T01:46:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_27 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_27 2024-09-18T01:46:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_27 2024-09-18T01:46:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-18T01:46:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_28 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_28 2024-09-18T01:46:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_28 2024-09-18T01:46:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-18T01:46:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-18T01:46:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-18T01:46:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_28 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_28 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_28 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-18T01:46:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_29 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:46:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_29 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_29 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-18T01:46:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-18T01:46:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-18T01:46:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_29 2024-09-18T01:46:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_29 2024-09-18T01:46:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_29 2024-09-18T01:46:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-18T01:46:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-18T01:46:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_30 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_30 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_30 2024-09-18T01:46:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-18T01:46:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-18T01:46:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-18T01:46:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_30 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_30 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_30 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-18T01:46:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_31 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:46:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_31 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_31 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-18T01:46:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-18T01:46:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-18T01:46:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_31 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_31 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_31 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-18T01:46:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-18T01:46:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_32 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_32 2024-09-18T01:46:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_32 2024-09-18T01:46:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-18T01:46:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-18T01:46:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-18T01:46:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_32 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_32 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_32 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-18T01:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-18T01:46:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_33 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_33 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_33 2024-09-18T01:46:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-18T01:46:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-18T01:46:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_33 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_33 2024-09-18T01:46:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_33 2024-09-18T01:46:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-18T01:46:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_34 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_34 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_34 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-18T01:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-18T01:46:13.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:13.968 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-18T01:46:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_34 2024-09-18T01:46:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_34 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_34 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-18T01:46:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-18T01:46:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_35 2024-09-18T01:46:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_35 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_35 2024-09-18T01:46:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-18T01:46:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-18T01:46:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-18T01:46:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_35 2024-09-18T01:46:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_35 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_35 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-18T01:46:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-18T01:46:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_36 2024-09-18T01:46:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-18T01:46:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_36 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_36 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-18T01:46:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-18T01:46:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-18T01:46:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_36 2024-09-18T01:46:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_36 2024-09-18T01:46:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_36 2024-09-18T01:46:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-18T01:46:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-18T01:46:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_37 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_37 2024-09-18T01:46:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_37 2024-09-18T01:46:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-18T01:46:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-18T01:46:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_37 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_37 2024-09-18T01:46:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_37 2024-09-18T01:46:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-18T01:46:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-18T01:46:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_38 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_38 2024-09-18T01:46:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_38 2024-09-18T01:46:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-18T01:46:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-18T01:46:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_38 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_38 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_38 2024-09-18T01:46:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-18T01:46:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_39 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_39 2024-09-18T01:46:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_39 2024-09-18T01:46:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-18T01:46:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-18T01:46:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-18T01:46:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_39 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_39 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_39 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-18T01:46:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-18T01:46:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_40 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_40 2024-09-18T01:46:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_40 2024-09-18T01:46:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-18T01:46:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-18T01:46:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-18T01:46:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_40 2024-09-18T01:46:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_40 2024-09-18T01:46:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_40 2024-09-18T01:46:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-18T01:46:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_41 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_41 2024-09-18T01:46:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_41 2024-09-18T01:46:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-18T01:46:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-18T01:46:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-18T01:46:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_41 2024-09-18T01:46:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_41 2024-09-18T01:46:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_41 2024-09-18T01:46:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-18T01:46:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_42 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_42 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_42 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-18T01:46:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-18T01:46:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_42 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_42 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_42 2024-09-18T01:46:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-18T01:46:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_43 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_43 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_43 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-18T01:46:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-18T01:46:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-18T01:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_43 2024-09-18T01:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_43 2024-09-18T01:46:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_43 2024-09-18T01:46:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-18T01:46:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-18T01:46:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_44 2024-09-18T01:46:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_44 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_44 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-18T01:46:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-18T01:46:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-18T01:46:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_44 2024-09-18T01:46:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_44 2024-09-18T01:46:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_44 2024-09-18T01:46:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-18T01:46:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-18T01:46:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_45 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_45 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_45 2024-09-18T01:46:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-18T01:46:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-18T01:46:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_45 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_45 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_45 2024-09-18T01:46:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-18T01:46:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-18T01:46:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_46 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_46 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_46 2024-09-18T01:46:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-18T01:46:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.086 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-18T01:46:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-18T01:46:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_46 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_46 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_46 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-18T01:46:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-18T01:46:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_47 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_47 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_47 2024-09-18T01:46:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-18T01:46:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-18T01:46:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_47 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_47 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_47 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-18T01:46:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-18T01:46:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_48 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_48 2024-09-18T01:46:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_48 2024-09-18T01:46:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-18T01:46:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-18T01:46:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-18T01:46:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_48 2024-09-18T01:46:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_48 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_48 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-18T01:46:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_49 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_49 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_49 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-18T01:46:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-18T01:46:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-18T01:46:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_49 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_49 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_49 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-18T01:46:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_50 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:46:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_50 2024-09-18T01:46:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_50 2024-09-18T01:46:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-18T01:46:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-18T01:46:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:46:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-18T01:46:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_50 2024-09-18T01:46:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:46:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_50 2024-09-18T01:46:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_50 2024-09-18T01:46:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-18T01:46:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-18T01:46:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:46:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:46:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-18T01:46:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ tee -a /tmp/tmp.YxZgdY10x3/rbd-mirror.test.log 2024-09-18T01:46:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ date '+%F %T' 2024-09-18T01:46:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo 2024-09-18 01:46:16 TEST: delete images during bootstrap 2024-09-18T01:46:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:2024-09-18 01:46:16 TEST: delete images during bootstrap 2024-09-18T01:46:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-18T01:46:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=image 2024-09-18T01:46:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-18T01:46:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd: mirroring is already configured for image mode 2024-09-18T01:46:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-18T01:46:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=image 2024-09-18T01:46:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-18T01:46:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd: mirroring is already configured for image mode 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ start_mirror cluster1 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local val=cluster1 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster_var_name=cluster 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local instance_var_name=instance 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cluster=cluster1 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=cluster1 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ instance=0 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval cluster=cluster1 2024-09-18T01:46:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ cluster=cluster1 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ eval instance=0 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ instance=0 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=test 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 1 10 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_1 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:46:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:46:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:46:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:46:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:46:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:46:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:46:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-18T01:46:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-18T01:46:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:46:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:46:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-18T01:46:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:46:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_1': 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 194097cc308b 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.194097cc308b 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:46:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:46:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:46:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:46:17 2024 2024-09-18T01:46:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:46:17 2024 2024-09-18T01:46:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:46:17 2024 2024-09-18T01:46:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:46:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:46:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 382e4dbc-4dc9-4c20-a484-662041b2e13a 2024-09-18T01:46:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:46:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-18T01:46:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:46:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:46:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=12 2024-09-18T01:46:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:46:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:46:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 50570240 bytes 2024-09-18T01:46:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 94512128 bytes 2024-09-18T01:46:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 181013504 bytes 2024-09-18T01:46:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 218278912 bytes 2024-09-18T01:46:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 245656576 bytes 2024-09-18T01:46:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 319388672 bytes 2024-09-18T01:46:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 345606144 bytes 2024-09-18T01:46:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 366643200 bytes 2024-09-18T01:46:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 404205568 bytes 2024-09-18T01:46:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 432899072 bytes 2024-09-18T01:46:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 480826368 bytes 2024-09-18T01:46:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 536116224 bytes 2024-09-18T01:46:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 561557504 bytes 2024-09-18T01:46:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 626914304 bytes 2024-09-18T01:46:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 633905152 bytes 2024-09-18T01:46:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 680396800 bytes 2024-09-18T01:46:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 681012224 bytes 2024-09-18T01:46:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 746914816 bytes 2024-09-18T01:46:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 784189440 bytes 2024-09-18T01:46:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 831011840 bytes 2024-09-18T01:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 910570496 bytes 2024-09-18T01:46:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 940406784 bytes 2024-09-18T01:46:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 977990656 bytes 2024-09-18T01:46:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 993485824 bytes 2024-09-18T01:46:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1022853120 bytes 2024-09-18T01:46:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1061619712 bytes 2024-09-18T01:46:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1095545856 bytes 2024-09-18T01:46:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1120188416 bytes 2024-09-18T01:46:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1172746240 bytes 2024-09-18T01:46:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1246277632 bytes 2024-09-18T01:46:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1280973824 bytes 2024-09-18T01:46:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1317113856 bytes 2024-09-18T01:46:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1354493952 bytes 2024-09-18T01:46:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1394193408 bytes 2024-09-18T01:46:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1419116544 bytes 2024-09-18T01:46:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1449271296 bytes 2024-09-18T01:46:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1475522560 bytes 2024-09-18T01:46:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1553729536 bytes 2024-09-18T01:46:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1593715712 bytes 2024-09-18T01:46:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1635988480 bytes 2024-09-18T01:46:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1660982272 bytes 2024-09-18T01:46:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1689984000 bytes 2024-09-18T01:46:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1709632512 bytes 2024-09-18T01:46:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1746191360 bytes 2024-09-18T01:46:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1780842496 bytes 2024-09-18T01:46:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1842396160 bytes 2024-09-18T01:46:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1881323520 bytes 2024-09-18T01:46:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1911485440 bytes 2024-09-18T01:46:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1953660928 bytes 2024-09-18T01:46:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 1985547264 bytes 2024-09-18T01:46:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2002170880 bytes 2024-09-18T01:46:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2077332480 bytes 2024-09-18T01:46:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2094185472 bytes 2024-09-18T01:46:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2101986304 bytes 2024-09-18T01:46:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2183769088 bytes 2024-09-18T01:46:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2275653632 bytes 2024-09-18T01:46:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2297345024 bytes 2024-09-18T01:46:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2357854208 bytes 2024-09-18T01:46:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2408463360 bytes 2024-09-18T01:46:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2480444416 bytes 2024-09-18T01:46:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2522454016 bytes 2024-09-18T01:46:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2523081728 bytes 2024-09-18T01:46:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2610019328 bytes 2024-09-18T01:46:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2650227712 bytes 2024-09-18T01:46:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2670948352 bytes 2024-09-18T01:46:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2707329024 bytes 2024-09-18T01:46:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2754272256 bytes 2024-09-18T01:46:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2778805248 bytes 2024-09-18T01:46:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2834246656 bytes 2024-09-18T01:46:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2924518400 bytes 2024-09-18T01:46:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3014336512 bytes 2024-09-18T01:46:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3095034880 bytes 2024-09-18T01:46:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3152756736 bytes 2024-09-18T01:46:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3205787648 bytes 2024-09-18T01:46:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3238845440 bytes 2024-09-18T01:46:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3302421504 bytes 2024-09-18T01:46:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3377830912 bytes 2024-09-18T01:46:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3421017088 bytes 2024-09-18T01:46:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3461120000 bytes 2024-09-18T01:46:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3464982528 bytes 2024-09-18T01:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3469355008 bytes 2024-09-18T01:46:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3578659840 bytes 2024-09-18T01:46:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3595660288 bytes 2024-09-18T01:46:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3644607488 bytes 2024-09-18T01:46:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3717031936 bytes 2024-09-18T01:46:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3783512064 bytes 2024-09-18T01:46:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3824756736 bytes 2024-09-18T01:46:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3866900480 bytes 2024-09-18T01:46:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3907629056 bytes 2024-09-18T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3949748224 bytes 2024-09-18T01:46:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3980270592 bytes 2024-09-18T01:46:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4019380224 bytes 2024-09-18T01:46:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4107920384 bytes 2024-09-18T01:46:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4137282560 bytes 2024-09-18T01:46:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4173661184 bytes 2024-09-18T01:46:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4193716224 bytes 2024-09-18T01:46:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4248489984 bytes 2024-09-18T01:46:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4252389376 bytes 2024-09-18T01:46:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4303560704 bytes 2024-09-18T01:46:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:46:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:46:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:46:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_1 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_1 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-18T01:46:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=14a7c358d162 2024-09-18T01:46:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:46:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:46:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_1 present 14a7c358d162 2024-09-18T01:46:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:46:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:46:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:46:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=14a7c358d162 2024-09-18T01:46:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:46:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:46:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_1 2024-09-18T01:46:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:46:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:46:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_1 2024-09-18T01:46:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-18T01:46:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:46:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=14a7c358d162 2024-09-18T01:46:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 14a7c358d162 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 14a7c358d162 -o 14a7c358d162 = 14a7c358d162 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:46:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-18T01:46:47.969 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.32858798986187643) > weight (0.25) 2024-09-18T01:46:47.969 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6087672068225171) > weight (0.25) 2024-09-18T01:46:47.969 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.35319260370290284) > weight (0.25) 2024-09-18T01:46:47.969 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-18T01:46:47.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-18T01:46:47.970 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-18T01:46:47.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:46:47.970+0000 7fe09ae5d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 52724) UID: 1000 2024-09-18T01:46:50.971 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-18T01:46:50.971 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:46:50.971 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-18T01:46:50.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-18T01:46:50.972 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-18T01:46:50.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-18T01:46:50.975 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-18T01:46:52.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:46:52.930+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecab6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:46:52.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:46:52.930+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f670c80 [3/382e4dbc-4dc9-4c20-a484-662041b2e13a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:46:55.763 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7683409341170201) > weight (0.25) 2024-09-18T01:46:55.764 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7018770776681499) > weight (0.25) 2024-09-18T01:46:55.764 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.633461075921849) > weight (0.25) 2024-09-18T01:46:55.764 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4324303673595644) > weight (0.25) 2024-09-18T01:46:55.764 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-18T01:47:00.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:47:00.229+0000 7f3f7a7e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a5f6569800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-09-18T01:47:00.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:47:00.229+0000 7f3f7a7e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a5f6569800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=909fcd80-1380-4b17-b581-43d9e4cb4eca, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-09-18T01:47:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:47:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-18T01:47:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_1 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:47:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_1 2024-09-18T01:47:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-18T01:47:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:47:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-18T01:47:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:47:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:47:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:47:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-18T01:47:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_1 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_1 2024-09-18T01:47:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-18T01:47:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:47:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-18T01:47:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:47:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:47:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:47:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:47:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-18T01:47:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:47:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-18T01:47:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-18T01:47:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:47:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:47:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:47:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_1 2024-09-18T01:47:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_1 2024-09-18T01:47:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-18T01:47:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-18T01:47:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:47:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:47:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:47:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_2 2024-09-18T01:47:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-18T01:47:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:47:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-18T01:47:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-18T01:47:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:47:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-18T01:47:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:47:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-18T01:47:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_2': 2024-09-18T01:47:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 196133f37e35 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.196133f37e35 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:47:00 2024 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:47:00 2024 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:47:00 2024 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:47:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: e4d92806-5dda-4351-b812-fbfe1ff9c7d6 2024-09-18T01:47:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:47:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-18T01:47:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:47:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:47:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:47:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:47:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 10530816 bytes 2024-09-18T01:47:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 64840704 bytes 2024-09-18T01:47:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 93306880 bytes 2024-09-18T01:47:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 113990656 bytes 2024-09-18T01:47:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 137321472 bytes 2024-09-18T01:47:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 166450176 bytes 2024-09-18T01:47:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 275684352 bytes 2024-09-18T01:47:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 294834176 bytes 2024-09-18T01:47:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 364108800 bytes 2024-09-18T01:47:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 440048640 bytes 2024-09-18T01:47:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 521864192 bytes 2024-09-18T01:47:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 539508736 bytes 2024-09-18T01:47:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 619202560 bytes 2024-09-18T01:47:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 647553024 bytes 2024-09-18T01:47:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 690702336 bytes 2024-09-18T01:47:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 698048512 bytes 2024-09-18T01:47:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 719283200 bytes 2024-09-18T01:47:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 756641792 bytes 2024-09-18T01:47:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 762589184 bytes 2024-09-18T01:47:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 806886400 bytes 2024-09-18T01:47:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 867846144 bytes 2024-09-18T01:47:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 904683520 bytes 2024-09-18T01:47:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1011348480 bytes 2024-09-18T01:47:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1040352256 bytes 2024-09-18T01:47:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1063932928 bytes 2024-09-18T01:47:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1078304768 bytes 2024-09-18T01:47:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1130008576 bytes 2024-09-18T01:47:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1230873600 bytes 2024-09-18T01:47:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1285640192 bytes 2024-09-18T01:47:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1343564800 bytes 2024-09-18T01:47:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1376544768 bytes 2024-09-18T01:47:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1411672064 bytes 2024-09-18T01:47:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1422090240 bytes 2024-09-18T01:47:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1471280128 bytes 2024-09-18T01:47:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1480192000 bytes 2024-09-18T01:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1574246400 bytes 2024-09-18T01:47:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1599648768 bytes 2024-09-18T01:47:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1667064832 bytes 2024-09-18T01:47:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1684161536 bytes 2024-09-18T01:47:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1748102144 bytes 2024-09-18T01:47:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1788002304 bytes 2024-09-18T01:47:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1808279552 bytes 2024-09-18T01:47:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1852853248 bytes 2024-09-18T01:47:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1875520512 bytes 2024-09-18T01:47:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1921633280 bytes 2024-09-18T01:47:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1974860800 bytes 2024-09-18T01:47:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2022641664 bytes 2024-09-18T01:47:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2112492544 bytes 2024-09-18T01:47:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2186752000 bytes 2024-09-18T01:47:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2243087360 bytes 2024-09-18T01:47:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2284652544 bytes 2024-09-18T01:47:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2314496000 bytes 2024-09-18T01:47:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2388955136 bytes 2024-09-18T01:47:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2525402112 bytes 2024-09-18T01:47:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2555935744 bytes 2024-09-18T01:47:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2605622272 bytes 2024-09-18T01:47:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2696737792 bytes 2024-09-18T01:47:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2741277696 bytes 2024-09-18T01:47:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2758102016 bytes 2024-09-18T01:47:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2797230080 bytes 2024-09-18T01:47:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2888457216 bytes 2024-09-18T01:47:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2939281408 bytes 2024-09-18T01:47:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2976137216 bytes 2024-09-18T01:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 3039789056 bytes 2024-09-18T01:47:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 3049065472 bytes 2024-09-18T01:47:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 3060051968 bytes 2024-09-18T01:47:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3091948544 bytes 2024-09-18T01:47:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3107860480 bytes 2024-09-18T01:47:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3131962368 bytes 2024-09-18T01:47:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3132587008 bytes 2024-09-18T01:47:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3161353216 bytes 2024-09-18T01:47:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3161958400 bytes 2024-09-18T01:47:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3208572928 bytes 2024-09-18T01:47:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:47:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:47:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:47:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:47:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_2 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:47:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:47:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_2 2024-09-18T01:47:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-18T01:47:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:47:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=14a7fca7f96f 2024-09-18T01:47:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:47:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:47:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_2 present 14a7fca7f96f 2024-09-18T01:47:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:47:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:47:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=14a7fca7f96f 2024-09-18T01:47:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:47:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:47:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_2 2024-09-18T01:47:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:47:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:47:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_2 2024-09-18T01:47:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-18T01:47:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:47:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=14a7fca7f96f 2024-09-18T01:47:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 14a7fca7f96f 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 14a7fca7f96f -o 14a7fca7f96f = 14a7fca7f96f 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-18T01:47:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:47:22.931+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ea0cea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:47:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:47:22.931+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f670c80 [3/e4d92806-5dda-4351-b812-fbfe1ff9c7d6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:47:40.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:47:40.285+0000 7f3f7a7e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a5f62af400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-18T01:47:40.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:47:40.285+0000 7f3f7a7e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a5f62af400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=909fcd80-1380-4b17-b581-43d9e4cb4eca, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-18T01:47:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:47:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-18T01:47:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:47:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_2 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_2 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-18T01:47:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:47:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-18T01:47:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:47:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:47:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_2 2024-09-18T01:47:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:47:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:47:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_2 2024-09-18T01:47:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-18T01:47:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:47:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-18T01:47:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:47:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:47:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:47:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:47:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-18T01:47:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-18T01:47:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-18T01:47:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:47:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:47:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_2 2024-09-18T01:47:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_2 2024-09-18T01:47:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-18T01:47:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-18T01:47:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:47:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:47:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_3 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:47:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-18T01:47:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-18T01:47:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:47:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:47:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-18T01:47:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:47:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-18T01:47:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_3': 2024-09-18T01:47:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:47:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:47:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:47:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 197932339661 2024-09-18T01:47:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.197932339661 2024-09-18T01:47:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:47:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:47:40 2024 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:47:40 2024 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:47:40 2024 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 7d933dfe-d6a2-4450-9706-c78e15d459e3 2024-09-18T01:47:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-18T01:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:47:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=7 2024-09-18T01:47:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:47:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:47:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 41571328 bytes 2024-09-18T01:47:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 71362560 bytes 2024-09-18T01:47:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 125530112 bytes 2024-09-18T01:47:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 176907264 bytes 2024-09-18T01:47:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 218821632 bytes 2024-09-18T01:47:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 220753920 bytes 2024-09-18T01:47:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 254382080 bytes 2024-09-18T01:47:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 254987264 bytes 2024-09-18T01:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 375480320 bytes 2024-09-18T01:47:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 401076224 bytes 2024-09-18T01:47:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 411141120 bytes 2024-09-18T01:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 432547840 bytes 2024-09-18T01:47:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 442961920 bytes 2024-09-18T01:47:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 473212928 bytes 2024-09-18T01:47:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 482727936 bytes 2024-09-18T01:47:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 523122688 bytes 2024-09-18T01:47:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 576472064 bytes 2024-09-18T01:47:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 590930944 bytes 2024-09-18T01:47:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 664978432 bytes 2024-09-18T01:47:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 745227264 bytes 2024-09-18T01:47:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 790224896 bytes 2024-09-18T01:47:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 842083328 bytes 2024-09-18T01:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 920165376 bytes 2024-09-18T01:47:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 973140992 bytes 2024-09-18T01:47:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1078574080 bytes 2024-09-18T01:47:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1105515520 bytes 2024-09-18T01:47:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1144008704 bytes 2024-09-18T01:47:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1190012928 bytes 2024-09-18T01:47:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1204939776 bytes 2024-09-18T01:47:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1226361856 bytes 2024-09-18T01:47:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1235435520 bytes 2024-09-18T01:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1242780672 bytes 2024-09-18T01:47:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1285211136 bytes 2024-09-18T01:47:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1345563648 bytes 2024-09-18T01:47:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1414196224 bytes 2024-09-18T01:47:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1510648832 bytes 2024-09-18T01:47:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1561908224 bytes 2024-09-18T01:47:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1564180480 bytes 2024-09-18T01:47:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1577340928 bytes 2024-09-18T01:47:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1666050048 bytes 2024-09-18T01:47:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1703553024 bytes 2024-09-18T01:47:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1773478912 bytes 2024-09-18T01:47:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1840721920 bytes 2024-09-18T01:47:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1873227776 bytes 2024-09-18T01:47:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1897593856 bytes 2024-09-18T01:47:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1930739712 bytes 2024-09-18T01:47:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1931338752 bytes 2024-09-18T01:47:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1989346304 bytes 2024-09-18T01:47:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2043388928 bytes 2024-09-18T01:47:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2079039488 bytes 2024-09-18T01:47:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2148259840 bytes 2024-09-18T01:47:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2181103616 bytes 2024-09-18T01:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2277550080 bytes 2024-09-18T01:47:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2322302976 bytes 2024-09-18T01:47:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2344561664 bytes 2024-09-18T01:47:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2391924736 bytes 2024-09-18T01:47:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:47:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:47:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:47:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:47:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-18T01:47:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:47:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:47:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:47:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:47:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:47:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:47:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_3 2024-09-18T01:47:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:47:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:47:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_3 2024-09-18T01:47:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-18T01:47:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:47:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=14a7fc65d212 2024-09-18T01:47:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:47:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:47:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_3 present 14a7fc65d212 2024-09-18T01:47:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:47:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=14a7fc65d212 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_3 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_3 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-18T01:47:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:47:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=14a7fc65d212 2024-09-18T01:47:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 14a7fc65d212 2024-09-18T01:47:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 14a7fc65d212 -o 14a7fc65d212 = 14a7fc65d212 2024-09-18T01:47:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:47:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:47:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-18T01:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-18T01:47:52.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:47:52.935+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecae340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:47:52.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:47:52.935+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f165900 [3/7d933dfe-d6a2-4450-9706-c78e15d459e3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:47:56.975 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-18T01:47:56.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-18T01:47:56.976 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-18T01:47:56.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:47:56.977+0000 7faef71de640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 61735) UID: 1000 2024-09-18T01:48:03.977 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-18T01:48:03.978 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:48:03.978 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-18T01:48:03.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-18T01:48:03.978 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-18T01:48:03.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-18T01:48:03.981 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-18T01:48:18.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:48:18.970+0000 7f3f7a7e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a5f554bc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-18T01:48:18.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:48:18.970+0000 7f3f7a7e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55a5f554bc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=909fcd80-1380-4b17-b581-43d9e4cb4eca, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-18T01:48:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:48:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-18T01:48:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_3 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_3 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-18T01:48:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:48:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-18T01:48:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:48:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:48:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:48:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_3 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_3 2024-09-18T01:48:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-18T01:48:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:48:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-18T01:48:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:48:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:48:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:48:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:48:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-18T01:48:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:48:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-18T01:48:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-18T01:48:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:48:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:48:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:48:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_3 2024-09-18T01:48:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_3 2024-09-18T01:48:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-18T01:48:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-18T01:48:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_4 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:48:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:48:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:48:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:48:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-18T01:48:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-18T01:48:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:48:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-18T01:48:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:48:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_4': 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 199a1b730fb3 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.199a1b730fb3 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:48:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:48:19 2024 2024-09-18T01:48:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:48:19 2024 2024-09-18T01:48:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:48:19 2024 2024-09-18T01:48:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:48:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:48:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 1a97aa66-3986-453e-90b3-429114cc06cc 2024-09-18T01:48:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-18T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:48:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=6 2024-09-18T01:48:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:48:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:48:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 33720320 bytes 2024-09-18T01:48:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 91652096 bytes 2024-09-18T01:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 142524416 bytes 2024-09-18T01:48:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 143129600 bytes 2024-09-18T01:48:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 171531264 bytes 2024-09-18T01:48:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 191719424 bytes 2024-09-18T01:48:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 288397312 bytes 2024-09-18T01:48:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 345043968 bytes 2024-09-18T01:48:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 377556992 bytes 2024-09-18T01:48:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 378133504 bytes 2024-09-18T01:48:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 440033280 bytes 2024-09-18T01:48:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 497037312 bytes 2024-09-18T01:48:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 541020160 bytes 2024-09-18T01:48:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 580738048 bytes 2024-09-18T01:48:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 602185728 bytes 2024-09-18T01:48:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 603355136 bytes 2024-09-18T01:48:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 664642560 bytes 2024-09-18T01:48:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 674970624 bytes 2024-09-18T01:48:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 706400256 bytes 2024-09-18T01:48:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 715065344 bytes 2024-09-18T01:48:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 727489536 bytes 2024-09-18T01:48:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 752868352 bytes 2024-09-18T01:48:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 779029504 bytes 2024-09-18T01:48:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 838084608 bytes 2024-09-18T01:48:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 859280384 bytes 2024-09-18T01:48:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 873227264 bytes 2024-09-18T01:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 952395776 bytes 2024-09-18T01:48:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 974077952 bytes 2024-09-18T01:48:23.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 980541440 bytes 2024-09-18T01:48:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 984229888 bytes 2024-09-18T01:48:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1046350848 bytes 2024-09-18T01:48:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1058886656 bytes 2024-09-18T01:48:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1125660672 bytes 2024-09-18T01:48:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1164570624 bytes 2024-09-18T01:48:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1178031104 bytes 2024-09-18T01:48:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1315350528 bytes 2024-09-18T01:48:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1416851456 bytes 2024-09-18T01:48:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1484438528 bytes 2024-09-18T01:48:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1498084352 bytes 2024-09-18T01:48:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1503746048 bytes 2024-09-18T01:48:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1512062976 bytes 2024-09-18T01:48:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1569261568 bytes 2024-09-18T01:48:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1588540416 bytes 2024-09-18T01:48:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1611245568 bytes 2024-09-18T01:48:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1633030144 bytes 2024-09-18T01:48:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1717976064 bytes 2024-09-18T01:48:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1754199040 bytes 2024-09-18T01:48:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 1800711168 bytes 2024-09-18T01:48:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 1846701056 bytes 2024-09-18T01:48:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:48:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:48:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:48:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:48:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-18T01:48:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:48:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:48:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:48:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:48:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:48:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_4 2024-09-18T01:48:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:48:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:48:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_4 2024-09-18T01:48:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-18T01:48:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:48:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=14a7a2ecf28e 2024-09-18T01:48:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:48:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_4 present 14a7a2ecf28e 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=14a7a2ecf28e 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_4 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:48:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:48:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_4 2024-09-18T01:48:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-18T01:48:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=14a7a2ecf28e 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 14a7a2ecf28e 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 14a7a2ecf28e -o 14a7a2ecf28e = 14a7a2ecf28e 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-18T01:48:52.764 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-18T01:48:52.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-18T01:48:52.765 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-18T01:48:52.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:48:52.766+0000 7f3f88002640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 57135) UID: 1000 2024-09-18T01:48:52.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:48:52.931+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecb0340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:48:52.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:48:52.931+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f165900 [3/1a97aa66-3986-453e-90b3-429114cc06cc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:48:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-18T01:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:48:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:48:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:48:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_4 2024-09-18T01:48:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:48:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:48:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_4 2024-09-18T01:48:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-18T01:48:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:48:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=14a7a2ecf28e 2024-09-18T01:48:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:48:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:48:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_4 deleted 14a7a2ecf28e 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=14a7a2ecf28e 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_4 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_4 2024-09-18T01:48:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-18T01:48:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:48:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=14a7a2ecf28e 2024-09-18T01:48:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 14a7a2ecf28e 2024-09-18T01:48:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 14a7a2ecf28e -o 14a7a2ecf28e = 14a7a2ecf28e 2024-09-18T01:48:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:48:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = present 2024-09-18T01:48:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:48:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_4 deleted 14a7a2ecf28e 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=14a7a2ecf28e 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_4 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_4 2024-09-18T01:48:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-18T01:48:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:48:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-18T01:48:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-18T01:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-18T01:48:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-18T01:48:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:48:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:48:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_4 2024-09-18T01:48:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_4 2024-09-18T01:48:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-18T01:48:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-18T01:48:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:48:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:48:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:48:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_5 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:48:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-18T01:48:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-18T01:48:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:48:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:48:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-18T01:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:48:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-18T01:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_5': 2024-09-18T01:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 19b274a19227 2024-09-18T01:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.19b274a19227 2024-09-18T01:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:48:57 2024 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:48:57 2024 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:48:57 2024 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: a6663cb7-78ac-4ebf-9b0d-183fcd347a49 2024-09-18T01:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:48:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-18T01:48:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:48:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:48:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:48:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:48:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=12 2024-09-18T01:48:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:48:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:48:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 19826688 bytes 2024-09-18T01:48:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 45533184 bytes 2024-09-18T01:48:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 52567040 bytes 2024-09-18T01:48:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 89341952 bytes 2024-09-18T01:48:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 142936064 bytes 2024-09-18T01:48:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 144067584 bytes 2024-09-18T01:48:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 226994176 bytes 2024-09-18T01:48:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 313760768 bytes 2024-09-18T01:48:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 389255168 bytes 2024-09-18T01:48:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 444094464 bytes 2024-09-18T01:48:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 488739840 bytes 2024-09-18T01:48:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 513071104 bytes 2024-09-18T01:48:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 546475008 bytes 2024-09-18T01:49:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 604426240 bytes 2024-09-18T01:49:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 640479232 bytes 2024-09-18T01:49:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 773053440 bytes 2024-09-18T01:49:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 773668864 bytes 2024-09-18T01:49:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 860945408 bytes 2024-09-18T01:49:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 887428096 bytes 2024-09-18T01:49:00.766 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-18T01:49:00.767 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:49:00.767 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-18T01:49:00.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-18T01:49:00.767 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-18T01:49:00.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-18T01:49:00.770 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-09-18T01:49:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 945233920 bytes 2024-09-18T01:49:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 983705600 bytes 2024-09-18T01:49:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1084132352 bytes 2024-09-18T01:49:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1133516800 bytes 2024-09-18T01:49:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1208716288 bytes 2024-09-18T01:49:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1241844736 bytes 2024-09-18T01:49:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1336134656 bytes 2024-09-18T01:49:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1350051840 bytes 2024-09-18T01:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1364371456 bytes 2024-09-18T01:49:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1450121216 bytes 2024-09-18T01:49:02.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1466047488 bytes 2024-09-18T01:49:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1496468480 bytes 2024-09-18T01:49:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1529807872 bytes 2024-09-18T01:49:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1607698432 bytes 2024-09-18T01:49:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1629139968 bytes 2024-09-18T01:49:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1700914176 bytes 2024-09-18T01:49:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1701520384 bytes 2024-09-18T01:49:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1767604224 bytes 2024-09-18T01:49:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1807825920 bytes 2024-09-18T01:49:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1878635520 bytes 2024-09-18T01:49:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1909274624 bytes 2024-09-18T01:49:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1971785728 bytes 2024-09-18T01:49:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 2010317824 bytes 2024-09-18T01:49:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2061942784 bytes 2024-09-18T01:49:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2129285120 bytes 2024-09-18T01:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2163262464 bytes 2024-09-18T01:49:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2178975744 bytes 2024-09-18T01:49:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2203242496 bytes 2024-09-18T01:49:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2223265792 bytes 2024-09-18T01:49:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2230977536 bytes 2024-09-18T01:49:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2271303680 bytes 2024-09-18T01:49:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2340619264 bytes 2024-09-18T01:49:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2345915392 bytes 2024-09-18T01:49:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2360221696 bytes 2024-09-18T01:49:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2368545792 bytes 2024-09-18T01:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2407031808 bytes 2024-09-18T01:49:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2416520192 bytes 2024-09-18T01:49:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2427834368 bytes 2024-09-18T01:49:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2478238720 bytes 2024-09-18T01:49:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2567101440 bytes 2024-09-18T01:49:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2611992576 bytes 2024-09-18T01:49:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2612621312 bytes 2024-09-18T01:49:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2683875328 bytes 2024-09-18T01:49:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2703233024 bytes 2024-09-18T01:49:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2723367936 bytes 2024-09-18T01:49:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2752510976 bytes 2024-09-18T01:49:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2782114816 bytes 2024-09-18T01:49:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2818419712 bytes 2024-09-18T01:49:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2834506752 bytes 2024-09-18T01:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2835136512 bytes 2024-09-18T01:49:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2873195520 bytes 2024-09-18T01:49:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 2876374016 bytes 2024-09-18T01:49:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 2947460096 bytes 2024-09-18T01:49:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 2996717568 bytes 2024-09-18T01:49:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3031028736 bytes 2024-09-18T01:49:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3065163776 bytes 2024-09-18T01:49:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3112509440 bytes 2024-09-18T01:49:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3142960128 bytes 2024-09-18T01:49:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3163634688 bytes 2024-09-18T01:49:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3200070656 bytes 2024-09-18T01:49:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3222431744 bytes 2024-09-18T01:49:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3259407360 bytes 2024-09-18T01:49:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3318281216 bytes 2024-09-18T01:49:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3391451136 bytes 2024-09-18T01:49:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3476294656 bytes 2024-09-18T01:49:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3520274432 bytes 2024-09-18T01:49:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3592725504 bytes 2024-09-18T01:49:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3637753856 bytes 2024-09-18T01:49:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3638387712 bytes 2024-09-18T01:49:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3661398016 bytes 2024-09-18T01:49:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3692016640 bytes 2024-09-18T01:49:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3757135872 bytes 2024-09-18T01:49:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3784059904 bytes 2024-09-18T01:49:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 3835610112 bytes 2024-09-18T01:49:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 3932096512 bytes 2024-09-18T01:49:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 3951248384 bytes 2024-09-18T01:49:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 3997624320 bytes 2024-09-18T01:49:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4048297984 bytes 2024-09-18T01:49:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4103678976 bytes 2024-09-18T01:49:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4169216000 bytes 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_5 2024-09-18T01:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_5 2024-09-18T01:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-18T01:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:49:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=15e690a4c29 2024-09-18T01:49:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:49:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_5 present 15e690a4c29 2024-09-18T01:49:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=15e690a4c29 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_5 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_5 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-18T01:49:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=15e690a4c29 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 15e690a4c29 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 15e690a4c29 -o 15e690a4c29 = 15e690a4c29 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:49:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:49:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-18T01:49:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:49:22.932+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eec2820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:49:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:49:22.932+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f165900 [3/a6663cb7-78ac-4ebf-9b0d-183fcd347a49] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:49:33.982 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9378567486407671) > weight (0.25) 2024-09-18T01:49:33.982 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8757426144686962) > weight (0.25) 2024-09-18T01:49:33.982 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7868766264611821) > weight (0.25) 2024-09-18T01:49:33.983 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5321861142085822) > weight (0.25) 2024-09-18T01:49:33.983 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-18T01:49:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:49:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_5 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:49:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:49:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_5 2024-09-18T01:49:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-18T01:49:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:49:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-18T01:49:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:49:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:49:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_5 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_5 2024-09-18T01:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-18T01:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:49:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-18T01:49:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:49:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:49:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:49:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:49:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-18T01:49:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:49:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:49:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-18T01:49:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-18T01:49:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:49:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:49:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:49:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:49:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_5 2024-09-18T01:49:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:49:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_5 2024-09-18T01:49:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-18T01:49:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-18T01:49:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:49:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:49:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:49:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_6 2024-09-18T01:49:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-18T01:49:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:49:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:49:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:49:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:49:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:49:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:49:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:49:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:49:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:49:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-18T01:49:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-18T01:49:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:49:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:49:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:49:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-18T01:49:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:49:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_6': 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 19cf4f1a7435 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.19cf4f1a7435 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:49:41 2024 2024-09-18T01:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:49:41 2024 2024-09-18T01:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:49:41 2024 2024-09-18T01:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 5c7fb348-b58e-4f81-817a-25a6b6c73702 2024-09-18T01:49:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-18T01:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:49:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:49:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=13 2024-09-18T01:49:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:49:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:49:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 619520 bytes 2024-09-18T01:49:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 53637120 bytes 2024-09-18T01:49:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 93366272 bytes 2024-09-18T01:49:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 129528832 bytes 2024-09-18T01:49:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 171702272 bytes 2024-09-18T01:49:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 199271424 bytes 2024-09-18T01:49:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 268487680 bytes 2024-09-18T01:49:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 325125120 bytes 2024-09-18T01:49:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 360752128 bytes 2024-09-18T01:49:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 421442560 bytes 2024-09-18T01:49:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 487586816 bytes 2024-09-18T01:49:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 518718464 bytes 2024-09-18T01:49:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 563218432 bytes 2024-09-18T01:49:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 567116800 bytes 2024-09-18T01:49:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 613057536 bytes 2024-09-18T01:49:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 627392512 bytes 2024-09-18T01:49:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 663394304 bytes 2024-09-18T01:49:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 694475776 bytes 2024-09-18T01:49:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 778659840 bytes 2024-09-18T01:49:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 823799808 bytes 2024-09-18T01:49:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 875140096 bytes 2024-09-18T01:49:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 898435072 bytes 2024-09-18T01:49:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 968961024 bytes 2024-09-18T01:49:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1047784448 bytes 2024-09-18T01:49:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1075186688 bytes 2024-09-18T01:49:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1136215040 bytes 2024-09-18T01:49:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1203701760 bytes 2024-09-18T01:49:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1240225792 bytes 2024-09-18T01:49:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1347062784 bytes 2024-09-18T01:49:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1358121984 bytes 2024-09-18T01:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1396707328 bytes 2024-09-18T01:49:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1411588096 bytes 2024-09-18T01:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1484076032 bytes 2024-09-18T01:49:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1523486720 bytes 2024-09-18T01:49:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1583273984 bytes 2024-09-18T01:49:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1603178496 bytes 2024-09-18T01:49:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1637920768 bytes 2024-09-18T01:49:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1641797632 bytes 2024-09-18T01:49:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1706206208 bytes 2024-09-18T01:49:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1728907264 bytes 2024-09-18T01:49:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1827345408 bytes 2024-09-18T01:49:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1894958080 bytes 2024-09-18T01:49:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1925181440 bytes 2024-09-18T01:49:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1956325376 bytes 2024-09-18T01:49:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2005855232 bytes 2024-09-18T01:49:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2028380160 bytes 2024-09-18T01:49:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2073311232 bytes 2024-09-18T01:49:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2151015424 bytes 2024-09-18T01:49:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2218848256 bytes 2024-09-18T01:49:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2270131200 bytes 2024-09-18T01:49:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2345698304 bytes 2024-09-18T01:49:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2416545792 bytes 2024-09-18T01:49:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2471305216 bytes 2024-09-18T01:49:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2497258496 bytes 2024-09-18T01:49:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2524633088 bytes 2024-09-18T01:49:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2531964928 bytes 2024-09-18T01:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2556433408 bytes 2024-09-18T01:49:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2580311040 bytes 2024-09-18T01:49:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2618723328 bytes 2024-09-18T01:49:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2632579072 bytes 2024-09-18T01:49:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2664143872 bytes 2024-09-18T01:49:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2763731968 bytes 2024-09-18T01:49:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2829021184 bytes 2024-09-18T01:49:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2855227392 bytes 2024-09-18T01:49:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2871112704 bytes 2024-09-18T01:49:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2901994496 bytes 2024-09-18T01:49:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2914973696 bytes 2024-09-18T01:49:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2922024960 bytes 2024-09-18T01:49:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2992527360 bytes 2024-09-18T01:49:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3056291840 bytes 2024-09-18T01:49:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3105569792 bytes 2024-09-18T01:49:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3157577728 bytes 2024-09-18T01:49:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3195558912 bytes 2024-09-18T01:49:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3277144064 bytes 2024-09-18T01:49:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3304801280 bytes 2024-09-18T01:49:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3340483584 bytes 2024-09-18T01:49:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3380629504 bytes 2024-09-18T01:49:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3403689984 bytes 2024-09-18T01:49:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3455172608 bytes 2024-09-18T01:49:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3499251712 bytes 2024-09-18T01:49:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3573318656 bytes 2024-09-18T01:49:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3596926976 bytes 2024-09-18T01:49:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3637047296 bytes 2024-09-18T01:49:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3662345216 bytes 2024-09-18T01:49:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3694003200 bytes 2024-09-18T01:49:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3715863552 bytes 2024-09-18T01:49:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3723348992 bytes 2024-09-18T01:49:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3815771136 bytes 2024-09-18T01:49:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3856631808 bytes 2024-09-18T01:49:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3903927296 bytes 2024-09-18T01:49:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 3908812800 bytes 2024-09-18T01:49:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 3938390016 bytes 2024-09-18T01:49:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4019643392 bytes 2024-09-18T01:49:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4032723968 bytes 2024-09-18T01:49:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4070611968 bytes 2024-09-18T01:49:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4081212416 bytes 2024-09-18T01:49:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4124891136 bytes 2024-09-18T01:49:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4160780288 bytes 2024-09-18T01:49:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4215868416 bytes 2024-09-18T01:49:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4254213120 bytes 2024-09-18T01:49:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4285532160 bytes 2024-09-18T01:49:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4286146560 bytes 2024-09-18T01:49:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4321316864 bytes 2024-09-18T01:49:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_6 2024-09-18T01:49:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_6 2024-09-18T01:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-18T01:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:49:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=15e6f5b38bc9 2024-09-18T01:49:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:49:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_6 present 15e6f5b38bc9 2024-09-18T01:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=15e6f5b38bc9 2024-09-18T01:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_6 2024-09-18T01:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_6 2024-09-18T01:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-18T01:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=15e6f5b38bc9 2024-09-18T01:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 15e6f5b38bc9 2024-09-18T01:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 15e6f5b38bc9 -o 15e6f5b38bc9 = 15e6f5b38bc9 2024-09-18T01:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:49:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-18T01:49:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:49:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:49:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:49:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-18T01:50:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:50:22.931+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec87860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:50:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:50:22.931+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f165900 [3/5c7fb348-b58e-4f81-817a-25a6b6c73702] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:50:25.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:50:25.294+0000 7f0dfa4f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4ced400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-18T01:50:25.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:50:25.294+0000 7f0dfa4f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4ced400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=192, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=909fcd80-1380-4b17-b581-43d9e4cb4eca, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-18T01:50:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-18T01:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_6 2024-09-18T01:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_6 2024-09-18T01:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-18T01:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:50:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-18T01:50:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:50:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:50:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:50:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-18T01:50:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:50:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_6 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_6 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-18T01:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:50:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-18T01:50:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:50:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:50:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:50:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:50:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-18T01:50:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:50:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:50:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-18T01:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-18T01:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:50:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_6 2024-09-18T01:50:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:50:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_6 2024-09-18T01:50:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-18T01:50:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-18T01:50:26.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:50:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:50:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:50:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_7 2024-09-18T01:50:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-18T01:50:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:50:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:50:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:50:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:50:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:50:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:50:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:50:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:50:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:50:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-18T01:50:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-18T01:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-18T01:50:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:50:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_7': 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 19e7557844f4 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.19e7557844f4 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:50:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:50:26 2024 2024-09-18T01:50:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:50:26 2024 2024-09-18T01:50:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:50:26 2024 2024-09-18T01:50:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:50:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:50:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 2727ced0-0def-40c3-a753-ce0e66ac943b 2024-09-18T01:50:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:50:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-18T01:50:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:50:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:50:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:50:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=5 2024-09-18T01:50:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:50:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:50:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 71456768 bytes 2024-09-18T01:50:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 103751680 bytes 2024-09-18T01:50:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 150652928 bytes 2024-09-18T01:50:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 198114304 bytes 2024-09-18T01:50:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 215500800 bytes 2024-09-18T01:50:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 265631744 bytes 2024-09-18T01:50:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 309806080 bytes 2024-09-18T01:50:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 410259456 bytes 2024-09-18T01:50:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 501705728 bytes 2024-09-18T01:50:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 516841472 bytes 2024-09-18T01:50:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 601281536 bytes 2024-09-18T01:50:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 643861504 bytes 2024-09-18T01:50:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 669485056 bytes 2024-09-18T01:50:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 719594496 bytes 2024-09-18T01:50:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 735002624 bytes 2024-09-18T01:50:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 781472768 bytes 2024-09-18T01:50:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 793015296 bytes 2024-09-18T01:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 833393664 bytes 2024-09-18T01:50:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 916041728 bytes 2024-09-18T01:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 975193088 bytes 2024-09-18T01:50:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 991340544 bytes 2024-09-18T01:50:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1047165952 bytes 2024-09-18T01:50:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1064857600 bytes 2024-09-18T01:50:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1094008832 bytes 2024-09-18T01:50:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1133816832 bytes 2024-09-18T01:50:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1154006016 bytes 2024-09-18T01:50:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1232776192 bytes 2024-09-18T01:50:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1283058688 bytes 2024-09-18T01:50:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1361855488 bytes 2024-09-18T01:50:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1372835840 bytes 2024-09-18T01:50:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1432581120 bytes 2024-09-18T01:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1444469760 bytes 2024-09-18T01:50:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1471104000 bytes 2024-09-18T01:50:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1492897792 bytes 2024-09-18T01:50:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1556381696 bytes 2024-09-18T01:50:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1629027328 bytes 2024-09-18T01:50:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1660077056 bytes 2024-09-18T01:50:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1730023424 bytes 2024-09-18T01:50:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1730664448 bytes 2024-09-18T01:50:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:50:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:50:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:50:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:50:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_7 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_7 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-18T01:50:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:50:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=15e679b6d926 2024-09-18T01:50:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:50:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:50:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_7 present 15e679b6d926 2024-09-18T01:50:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:50:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:50:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:50:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=15e679b6d926 2024-09-18T01:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_7 2024-09-18T01:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_7 2024-09-18T01:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-18T01:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:50:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=15e679b6d926 2024-09-18T01:50:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 15e679b6d926 2024-09-18T01:50:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 15e679b6d926 -o 15e679b6d926 = 15e679b6d926 2024-09-18T01:50:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:50:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:50:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:50:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-18T01:50:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:50:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:50:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:50:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-18T01:50:45.771 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8219995864797535) > weight (0.25) 2024-09-18T01:50:45.771 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9707844271753516) > weight (0.25) 2024-09-18T01:50:45.771 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.45026390784881043) > weight (0.25) 2024-09-18T01:50:45.771 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.26822493065155084) > weight (0.25) 2024-09-18T01:50:45.771 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-09-18T01:50:52.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:50:52.932+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083eec4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:50:52.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:50:52.932+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f165180 [3/2727ced0-0def-40c3-a753-ce0e66ac943b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:50:57.984 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-18T01:50:57.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-18T01:50:57.985 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-18T01:50:57.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:50:57.985+0000 7f7e315f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 69563) UID: 1000 2024-09-18T01:51:01.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:51:01.250+0000 7f0df9cf5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4cdf000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2024-09-18T01:51:01.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:51:01.250+0000 7f0df9cf5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4cdf000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=909fcd80-1380-4b17-b581-43d9e4cb4eca, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2024-09-18T01:51:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:51:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-18T01:51:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:51:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:51:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:51:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:51:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:51:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:51:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_7 2024-09-18T01:51:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:51:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:51:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_7 2024-09-18T01:51:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-18T01:51:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:51:01.320 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-18T01:51:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:51:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:51:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:51:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_7 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_7 2024-09-18T01:51:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-18T01:51:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:51:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-18T01:51:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:51:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:51:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:51:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:51:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-18T01:51:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:51:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-18T01:51:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-18T01:51:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:51:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:51:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:51:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_7 2024-09-18T01:51:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_7 2024-09-18T01:51:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-18T01:51:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-18T01:51:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_8 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:51:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:51:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:51:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:51:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:51:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-18T01:51:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-18T01:51:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:51:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-18T01:51:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:51:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-18T01:51:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_8': 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 19ff4530445d 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.19ff4530445d 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:51:01 2024 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:51:01 2024 2024-09-18T01:51:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:51:01 2024 2024-09-18T01:51:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:51:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:51:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 38267e72-d254-4e65-88cd-ec879a115aaa 2024-09-18T01:51:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:51:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-18T01:51:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:51:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=14 2024-09-18T01:51:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:51:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:51:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 15213568 bytes 2024-09-18T01:51:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 52062208 bytes 2024-09-18T01:51:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 74427392 bytes 2024-09-18T01:51:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 145970176 bytes 2024-09-18T01:51:03.038 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 150004736 bytes 2024-09-18T01:51:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 199084032 bytes 2024-09-18T01:51:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 310139904 bytes 2024-09-18T01:51:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 395532288 bytes 2024-09-18T01:51:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 455304192 bytes 2024-09-18T01:51:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 509520896 bytes 2024-09-18T01:51:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 597681152 bytes 2024-09-18T01:51:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 649355264 bytes 2024-09-18T01:51:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 655449088 bytes 2024-09-18T01:51:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 705542144 bytes 2024-09-18T01:51:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 729511936 bytes 2024-09-18T01:51:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 778254336 bytes 2024-09-18T01:51:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 806750208 bytes 2024-09-18T01:51:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 808886272 bytes 2024-09-18T01:51:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 809511936 bytes 2024-09-18T01:51:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 831291392 bytes 2024-09-18T01:51:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 846484480 bytes 2024-09-18T01:51:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 847092736 bytes 2024-09-18T01:51:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 929041408 bytes 2024-09-18T01:51:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 945627136 bytes 2024-09-18T01:51:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1004111872 bytes 2024-09-18T01:51:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1047142400 bytes 2024-09-18T01:51:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1081662464 bytes 2024-09-18T01:51:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1097322496 bytes 2024-09-18T01:51:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1152457728 bytes 2024-09-18T01:51:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1238627328 bytes 2024-09-18T01:51:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1270179840 bytes 2024-09-18T01:51:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1324930048 bytes 2024-09-18T01:51:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1387478016 bytes 2024-09-18T01:51:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1463968768 bytes 2024-09-18T01:51:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1512504320 bytes 2024-09-18T01:51:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1588769792 bytes 2024-09-18T01:51:06.985 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-18T01:51:06.985 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:51:06.986 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-18T01:51:06.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-18T01:51:06.986 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-18T01:51:06.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-18T01:51:06.989 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-09-18T01:51:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1607242752 bytes 2024-09-18T01:51:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1653407744 bytes 2024-09-18T01:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1683955712 bytes 2024-09-18T01:51:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1725577216 bytes 2024-09-18T01:51:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1753372672 bytes 2024-09-18T01:51:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1772162048 bytes 2024-09-18T01:51:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1797258240 bytes 2024-09-18T01:51:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1842753536 bytes 2024-09-18T01:51:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1884854272 bytes 2024-09-18T01:51:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 1923860480 bytes 2024-09-18T01:51:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 1982840832 bytes 2024-09-18T01:51:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2012566528 bytes 2024-09-18T01:51:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2029459456 bytes 2024-09-18T01:51:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2105980928 bytes 2024-09-18T01:51:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2128008192 bytes 2024-09-18T01:51:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2144010240 bytes 2024-09-18T01:51:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2176598016 bytes 2024-09-18T01:51:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2290045952 bytes 2024-09-18T01:51:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2316777472 bytes 2024-09-18T01:51:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2323895296 bytes 2024-09-18T01:51:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2357964800 bytes 2024-09-18T01:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2411677696 bytes 2024-09-18T01:51:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2470263808 bytes 2024-09-18T01:51:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2480190464 bytes 2024-09-18T01:51:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2530468864 bytes 2024-09-18T01:51:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2531101696 bytes 2024-09-18T01:51:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2545511424 bytes 2024-09-18T01:51:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2565534720 bytes 2024-09-18T01:51:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2637771776 bytes 2024-09-18T01:51:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2717186048 bytes 2024-09-18T01:51:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2821561344 bytes 2024-09-18T01:51:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2922618880 bytes 2024-09-18T01:51:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 2974945280 bytes 2024-09-18T01:51:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 2977280000 bytes 2024-09-18T01:51:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3042546688 bytes 2024-09-18T01:51:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3087891456 bytes 2024-09-18T01:51:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3144941568 bytes 2024-09-18T01:51:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3200679936 bytes 2024-09-18T01:51:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3259997184 bytes 2024-09-18T01:51:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3312052224 bytes 2024-09-18T01:51:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3370354688 bytes 2024-09-18T01:51:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3402389504 bytes 2024-09-18T01:51:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3444362240 bytes 2024-09-18T01:51:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3501615104 bytes 2024-09-18T01:51:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3517289472 bytes 2024-09-18T01:51:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3592690688 bytes 2024-09-18T01:51:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3635828736 bytes 2024-09-18T01:51:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3708042240 bytes 2024-09-18T01:51:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3733143552 bytes 2024-09-18T01:51:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3803164672 bytes 2024-09-18T01:51:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3867121664 bytes 2024-09-18T01:51:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3966320640 bytes 2024-09-18T01:51:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 4013061120 bytes 2024-09-18T01:51:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 4051472384 bytes 2024-09-18T01:51:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4057074688 bytes 2024-09-18T01:51:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4057682944 bytes 2024-09-18T01:51:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4095551488 bytes 2024-09-18T01:51:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4151255040 bytes 2024-09-18T01:51:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4199673856 bytes 2024-09-18T01:51:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4256263168 bytes 2024-09-18T01:51:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4284825600 bytes 2024-09-18T01:51:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4298543104 bytes 2024-09-18T01:51:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4335514624 bytes 2024-09-18T01:51:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4375042048 bytes 2024-09-18T01:51:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4469240832 bytes 2024-09-18T01:51:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4507068416 bytes 2024-09-18T01:51:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4561159168 bytes 2024-09-18T01:51:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 4653590528 bytes 2024-09-18T01:51:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4685395968 bytes 2024-09-18T01:51:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 4738284544 bytes 2024-09-18T01:51:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 4787826688 bytes 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_8 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_8 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-18T01:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:51:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=15e61e65844d 2024-09-18T01:51:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:51:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_8 present 15e61e65844d 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=15e61e65844d 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_8 2024-09-18T01:51:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:51:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:51:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_8 2024-09-18T01:51:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-18T01:51:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=15e61e65844d 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 15e61e65844d 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 15e61e65844d -o 15e61e65844d = 15e61e65844d 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-18T01:51:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:51:22.932+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ec87520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:51:22.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:51:22.932+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f165180 [3/38267e72-d254-4e65-88cd-ec879a115aaa] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:51:47.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:51:47.299+0000 7f0df9cf5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4aa1000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=192 2024-09-18T01:51:47.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:51:47.299+0000 7f0df9cf5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4aa1000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=909fcd80-1380-4b17-b581-43d9e4cb4eca, primary_snap_id=c0, last_copied_object_number=128, snap_seqs={192=18446744073709551614}] 2024-09-18T01:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:51:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-18T01:51:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:51:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_8 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:51:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_8 2024-09-18T01:51:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-18T01:51:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:51:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-18T01:51:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:51:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:51:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:51:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-18T01:51:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:51:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:51:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:51:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:51:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:51:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_8 2024-09-18T01:51:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:51:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:51:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_8 2024-09-18T01:51:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-18T01:51:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:51:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-18T01:51:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:51:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:51:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:51:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:51:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-18T01:51:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-18T01:51:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-18T01:51:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:51:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:51:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_8 2024-09-18T01:51:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_8 2024-09-18T01:51:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-18T01:51:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-18T01:51:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:51:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:51:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:51:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_9 2024-09-18T01:51:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-18T01:51:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-18T01:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-18T01:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:51:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:51:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-18T01:51:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:51:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-18T01:51:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_9': 2024-09-18T01:51:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:51:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:51:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:51:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 1a2133009a5e 2024-09-18T01:51:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.1a2133009a5e 2024-09-18T01:51:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:51:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:51:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:51:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:51:47 2024 2024-09-18T01:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:51:47 2024 2024-09-18T01:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:51:47 2024 2024-09-18T01:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 9c2020be-ba76-4dbe-975e-1e2e3423893b 2024-09-18T01:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:51:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-18T01:51:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:51:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:51:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=9 2024-09-18T01:51:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:51:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:51:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 50618368 bytes 2024-09-18T01:51:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 85083136 bytes 2024-09-18T01:51:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 189069312 bytes 2024-09-18T01:51:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 275796992 bytes 2024-09-18T01:51:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 309196800 bytes 2024-09-18T01:51:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 385925120 bytes 2024-09-18T01:51:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 429460480 bytes 2024-09-18T01:51:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 462030848 bytes 2024-09-18T01:51:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 467256320 bytes 2024-09-18T01:51:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 495834112 bytes 2024-09-18T01:51:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 527878144 bytes 2024-09-18T01:51:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 616718336 bytes 2024-09-18T01:51:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 711462912 bytes 2024-09-18T01:51:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 712065024 bytes 2024-09-18T01:51:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 796879872 bytes 2024-09-18T01:51:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 864737280 bytes 2024-09-18T01:51:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 928945152 bytes 2024-09-18T01:51:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 934788096 bytes 2024-09-18T01:51:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 959199232 bytes 2024-09-18T01:51:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 969263104 bytes 2024-09-18T01:51:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 1013814272 bytes 2024-09-18T01:51:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 1047137280 bytes 2024-09-18T01:51:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1069512704 bytes 2024-09-18T01:51:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1156359168 bytes 2024-09-18T01:51:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1215593472 bytes 2024-09-18T01:51:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1259113472 bytes 2024-09-18T01:51:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1283091456 bytes 2024-09-18T01:51:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1339214848 bytes 2024-09-18T01:51:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1381434368 bytes 2024-09-18T01:51:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1397360640 bytes 2024-09-18T01:51:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1433104384 bytes 2024-09-18T01:51:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1495249920 bytes 2024-09-18T01:51:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1527582720 bytes 2024-09-18T01:51:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1570025472 bytes 2024-09-18T01:51:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1633738752 bytes 2024-09-18T01:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1750071296 bytes 2024-09-18T01:51:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1769797632 bytes 2024-09-18T01:51:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1793977344 bytes 2024-09-18T01:51:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1846967296 bytes 2024-09-18T01:51:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1901595648 bytes 2024-09-18T01:51:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1902213120 bytes 2024-09-18T01:51:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1963742208 bytes 2024-09-18T01:51:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 2037086208 bytes 2024-09-18T01:51:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 2041928704 bytes 2024-09-18T01:51:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 2091930624 bytes 2024-09-18T01:51:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2146527232 bytes 2024-09-18T01:51:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2167964672 bytes 2024-09-18T01:51:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2208438272 bytes 2024-09-18T01:51:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2227365888 bytes 2024-09-18T01:51:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2285984768 bytes 2024-09-18T01:51:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2344993792 bytes 2024-09-18T01:51:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2348109824 bytes 2024-09-18T01:51:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2363689984 bytes 2024-09-18T01:51:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2408419328 bytes 2024-09-18T01:51:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2530139136 bytes 2024-09-18T01:51:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2537798656 bytes 2024-09-18T01:51:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2593913856 bytes 2024-09-18T01:51:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2623763456 bytes 2024-09-18T01:51:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2658467840 bytes 2024-09-18T01:51:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2702597120 bytes 2024-09-18T01:51:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2726423552 bytes 2024-09-18T01:51:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2744480768 bytes 2024-09-18T01:51:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2770021376 bytes 2024-09-18T01:51:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2819704832 bytes 2024-09-18T01:51:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2854862848 bytes 2024-09-18T01:51:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2896880640 bytes 2024-09-18T01:51:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 2897510400 bytes 2024-09-18T01:51:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 2990409728 bytes 2024-09-18T01:51:56.883 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3028054016 bytes 2024-09-18T01:51:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3042777088 bytes 2024-09-18T01:51:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3135292416 bytes 2024-09-18T01:51:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3141224448 bytes 2024-09-18T01:51:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3160980480 bytes 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:51:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:51:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_9 2024-09-18T01:51:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:51:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:51:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_9 2024-09-18T01:51:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-18T01:51:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:51:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=15e66f8c9cb6 2024-09-18T01:51:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:51:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:51:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_9 present 15e66f8c9cb6 2024-09-18T01:51:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:51:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:51:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=15e66f8c9cb6 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_9 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_9 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-18T01:51:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=15e66f8c9cb6 2024-09-18T01:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 15e66f8c9cb6 2024-09-18T01:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 15e66f8c9cb6 -o 15e66f8c9cb6 = 15e66f8c9cb6 2024-09-18T01:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-18T01:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-18T01:52:18.772 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5420117635132743) > weight (0.25) 2024-09-18T01:52:18.772 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-18T01:52:18.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-18T01:52:18.773 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-18T01:52:18.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:52:18.773+0000 7f1c962bc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 59526) UID: 1000 2024-09-18T01:52:22.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:52:22.933+0000 7f2152d00640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083e7f4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:52:22.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:52:22.933+0000 7f215950d640 -1 rbd::mirror::ImageReplayer: 0x56083f165180 [3/9c2020be-ba76-4dbe-975e-1e2e3423893b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:52:27.774 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-18T01:52:27.774 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:52:27.774 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-18T01:52:27.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-18T01:52:27.775 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-18T01:52:27.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-18T01:52:27.778 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-18T01:52:29.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:52:29.067+0000 7f0dfa4f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4cdf400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2024-09-18T01:52:29.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:52:29.067+0000 7f0dfa4f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4cdf400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=195, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=909fcd80-1380-4b17-b581-43d9e4cb4eca, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2024-09-18T01:52:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:52:29.082 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:52:29.081+0000 7f78fa24b640 -1 reset not still connected to 0x55b9c5f9c000 2024-09-18T01:52:29.082 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:52:29.081+0000 7f78fa24b640 -1 reset not still connected to 0x55b9c6a008f0 2024-09-18T01:52:29.082 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:52:29.081+0000 7f78fa24b640 -1 reset not still connected to 0x55b9c8bb4820 2024-09-18T01:52:29.082 INFO:tasks.ceph.cluster2.osd.2.smithi115.stderr:2024-09-18T01:52:29.081+0000 7f78fa24b640 -1 reset not still connected to 0x55b9de7ed790 2024-09-18T01:52:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-18T01:52:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:52:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_9 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_9 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-18T01:52:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:52:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-18T01:52:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:52:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:52:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:52:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_9 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:52:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_9 2024-09-18T01:52:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-18T01:52:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:52:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-18T01:52:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:52:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:52:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:52:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:52:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-18T01:52:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:52:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:52:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-18T01:52:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-18T01:52:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:52:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:52:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:52:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:52:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_9 2024-09-18T01:52:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:52:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_9 2024-09-18T01:52:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-18T01:52:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-18T01:52:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:52:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:52:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for i in `seq 1 10` 2024-09-18T01:52:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image=image_10 2024-09-18T01:52:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-18T01:52:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:52:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:52:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:52:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:52:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:52:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local size=128 2024-09-18T01:52:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -n 512M ']' 2024-09-18T01:52:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ size=512M 2024-09-18T01:52:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ shift 2024-09-18T01:52:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-18T01:52:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-18T01:52:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:52:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:52:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local mode=snapshot 2024-09-18T01:52:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-18T01:52:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring enabled 2024-09-18T01:52:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:rbd image 'image_10': 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: size 512 MiB in 128 objects 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: snapshot_count: 1 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: id: 1a3fc4c35baf 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: block_name_prefix: rbd_data.1a3fc4c35baf 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: format: 2 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T01:52:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: op_features: 2024-09-18T01:52:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: flags: 2024-09-18T01:52:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: create_timestamp: Wed Sep 18 01:52:29 2024 2024-09-18T01:52:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: access_timestamp: Wed Sep 18 01:52:29 2024 2024-09-18T01:52:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: modify_timestamp: Wed Sep 18 01:52:29 2024 2024-09-18T01:52:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring state: enabled 2024-09-18T01:52:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring mode: snapshot 2024-09-18T01:52:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring global id: 3e63fc82-3737-4ce5-ae5a-0f487fbe3a72 2024-09-18T01:52:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout: mirroring primary: true 2024-09-18T01:52:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-18T01:52:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:52:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:52:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-18T01:52:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local duration=14 2024-09-18T01:52:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:52:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-18T01:52:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:100 IOs, 45791232 bytes 2024-09-18T01:52:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:200 IOs, 94248960 bytes 2024-09-18T01:52:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:300 IOs, 99663872 bytes 2024-09-18T01:52:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:400 IOs, 100281344 bytes 2024-09-18T01:52:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:500 IOs, 144338944 bytes 2024-09-18T01:52:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:600 IOs, 196990976 bytes 2024-09-18T01:52:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:700 IOs, 252622848 bytes 2024-09-18T01:52:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:800 IOs, 306056192 bytes 2024-09-18T01:52:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:900 IOs, 324043776 bytes 2024-09-18T01:52:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1000 IOs, 358251520 bytes 2024-09-18T01:52:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1100 IOs, 382416896 bytes 2024-09-18T01:52:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1200 IOs, 469595136 bytes 2024-09-18T01:52:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1300 IOs, 542405632 bytes 2024-09-18T01:52:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1400 IOs, 597827584 bytes 2024-09-18T01:52:32.101 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1500 IOs, 653217792 bytes 2024-09-18T01:52:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1600 IOs, 679372800 bytes 2024-09-18T01:52:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1700 IOs, 732587008 bytes 2024-09-18T01:52:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1800 IOs, 832121856 bytes 2024-09-18T01:52:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:1900 IOs, 855095296 bytes 2024-09-18T01:52:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2000 IOs, 877139968 bytes 2024-09-18T01:52:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2100 IOs, 880902144 bytes 2024-09-18T01:52:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2200 IOs, 943849472 bytes 2024-09-18T01:52:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2300 IOs, 1038182400 bytes 2024-09-18T01:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2400 IOs, 1157833728 bytes 2024-09-18T01:52:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2500 IOs, 1194350592 bytes 2024-09-18T01:52:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2600 IOs, 1212182528 bytes 2024-09-18T01:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2700 IOs, 1222583296 bytes 2024-09-18T01:52:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2800 IOs, 1284549632 bytes 2024-09-18T01:52:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:2900 IOs, 1323001856 bytes 2024-09-18T01:52:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3000 IOs, 1329297408 bytes 2024-09-18T01:52:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3100 IOs, 1383560192 bytes 2024-09-18T01:52:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3200 IOs, 1449255936 bytes 2024-09-18T01:52:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3300 IOs, 1478804480 bytes 2024-09-18T01:52:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3400 IOs, 1524046848 bytes 2024-09-18T01:52:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3500 IOs, 1594374144 bytes 2024-09-18T01:52:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3600 IOs, 1623203840 bytes 2024-09-18T01:52:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3700 IOs, 1712442368 bytes 2024-09-18T01:52:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3800 IOs, 1739585536 bytes 2024-09-18T01:52:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:3900 IOs, 1795353600 bytes 2024-09-18T01:52:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4000 IOs, 1830160384 bytes 2024-09-18T01:52:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4100 IOs, 1834849280 bytes 2024-09-18T01:52:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4200 IOs, 1869221888 bytes 2024-09-18T01:52:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4300 IOs, 1891420160 bytes 2024-09-18T01:52:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4400 IOs, 1915629568 bytes 2024-09-18T01:52:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4500 IOs, 1983633408 bytes 2024-09-18T01:52:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4600 IOs, 2019487744 bytes 2024-09-18T01:52:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4700 IOs, 2098532352 bytes 2024-09-18T01:52:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4800 IOs, 2139267072 bytes 2024-09-18T01:52:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:4900 IOs, 2192782336 bytes 2024-09-18T01:52:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5000 IOs, 2208010240 bytes 2024-09-18T01:52:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5100 IOs, 2257805312 bytes 2024-09-18T01:52:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5200 IOs, 2316371968 bytes 2024-09-18T01:52:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5300 IOs, 2345792512 bytes 2024-09-18T01:52:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5400 IOs, 2434663424 bytes 2024-09-18T01:52:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5500 IOs, 2446058496 bytes 2024-09-18T01:52:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5600 IOs, 2479149056 bytes 2024-09-18T01:52:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5700 IOs, 2499603456 bytes 2024-09-18T01:52:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5800 IOs, 2570559488 bytes 2024-09-18T01:52:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:5900 IOs, 2631455744 bytes 2024-09-18T01:52:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6000 IOs, 2645465088 bytes 2024-09-18T01:52:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6100 IOs, 2700150784 bytes 2024-09-18T01:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6200 IOs, 2740946944 bytes 2024-09-18T01:52:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6300 IOs, 2787239936 bytes 2024-09-18T01:52:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6400 IOs, 2840558592 bytes 2024-09-18T01:52:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6500 IOs, 2930938880 bytes 2024-09-18T01:52:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6600 IOs, 2996349952 bytes 2024-09-18T01:52:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6700 IOs, 3014266880 bytes 2024-09-18T01:52:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6800 IOs, 3064200192 bytes 2024-09-18T01:52:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:6900 IOs, 3141445632 bytes 2024-09-18T01:52:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7000 IOs, 3207465984 bytes 2024-09-18T01:52:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7100 IOs, 3208072192 bytes 2024-09-18T01:52:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7200 IOs, 3233906688 bytes 2024-09-18T01:52:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7300 IOs, 3259198464 bytes 2024-09-18T01:52:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7400 IOs, 3320639488 bytes 2024-09-18T01:52:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7500 IOs, 3334407168 bytes 2024-09-18T01:52:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7600 IOs, 3368139776 bytes 2024-09-18T01:52:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7700 IOs, 3380608000 bytes 2024-09-18T01:52:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7800 IOs, 3429309440 bytes 2024-09-18T01:52:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:7900 IOs, 3516948480 bytes 2024-09-18T01:52:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8000 IOs, 3524579328 bytes 2024-09-18T01:52:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8100 IOs, 3578086400 bytes 2024-09-18T01:52:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8200 IOs, 3670841344 bytes 2024-09-18T01:52:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8300 IOs, 3714321408 bytes 2024-09-18T01:52:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8400 IOs, 3760070656 bytes 2024-09-18T01:52:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8500 IOs, 3786749952 bytes 2024-09-18T01:52:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8600 IOs, 3829680128 bytes 2024-09-18T01:52:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8700 IOs, 3870909440 bytes 2024-09-18T01:52:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8800 IOs, 3939485696 bytes 2024-09-18T01:52:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:8900 IOs, 3962130432 bytes 2024-09-18T01:52:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9000 IOs, 3987536896 bytes 2024-09-18T01:52:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9100 IOs, 4056344576 bytes 2024-09-18T01:52:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9200 IOs, 4094200832 bytes 2024-09-18T01:52:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9300 IOs, 4129317888 bytes 2024-09-18T01:52:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9400 IOs, 4137448448 bytes 2024-09-18T01:52:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9500 IOs, 4151504896 bytes 2024-09-18T01:52:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9600 IOs, 4160118784 bytes 2024-09-18T01:52:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9700 IOs, 4173090816 bytes 2024-09-18T01:52:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9800 IOs, 4238795776 bytes 2024-09-18T01:52:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:9900 IOs, 4268222464 bytes 2024-09-18T01:52:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10000 IOs, 4298357760 bytes 2024-09-18T01:52:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10100 IOs, 4331085824 bytes 2024-09-18T01:52:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10200 IOs, 4365747200 bytes 2024-09-18T01:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10300 IOs, 4410603520 bytes 2024-09-18T01:52:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10400 IOs, 4453389312 bytes 2024-09-18T01:52:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10500 IOs, 4505366528 bytes 2024-09-18T01:52:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10600 IOs, 4555804672 bytes 2024-09-18T01:52:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10700 IOs, 4653516800 bytes 2024-09-18T01:52:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10800 IOs, 4681123840 bytes 2024-09-18T01:52:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:10900 IOs, 4728001536 bytes 2024-09-18T01:52:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11000 IOs, 4770173952 bytes 2024-09-18T01:52:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11100 IOs, 4803237888 bytes 2024-09-18T01:52:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:11200 IOs, 4838152192 bytes 2024-09-18T01:52:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=124 2024-09-18T01:52:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:52:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 124 -eq 124 ']' 2024-09-18T01:52:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:52:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-18T01:52:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:52:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:52:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=present 2024-09-18T01:52:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:52:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:52:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:52:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_10 2024-09-18T01:52:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:52:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:52:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_10 2024-09-18T01:52:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-18T01:52:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:52:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id=15e67119b757 2024-09-18T01:52:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:52:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:52:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_10 present 15e67119b757 2024-09-18T01:52:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:52:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=present 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id=15e67119b757 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_10 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_10 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-18T01:52:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id=15e67119b757 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 15e67119b757 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -z 15e67119b757 -o 15e67119b757 = 15e67119b757 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_state=present 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test present = present 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:52:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-18T01:52:52.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:52:52.932+0000 7f2153501640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56083ecaa820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-18T01:52:52.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:52:52.932+0000 7f2159d0e640 -1 rbd::mirror::ImageReplayer: 0x56083f6b3680 [3/3e63fc82-3737-4ce5-ae5a-0f487fbe3a72] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-18T01:53:05.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5369500899490672) > weight (0.25) 2024-09-18T01:53:05.990 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-18T01:53:05.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-18T01:53:05.991 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-18T01:53:05.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:05.991+0000 7f4e28cbd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 65585) UID: 1000 2024-09-18T01:53:10.992 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-18T01:53:10.992 INFO:teuthology.orchestra.run:waiting for 600 2024-09-18T01:53:10.993 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-18T01:53:10.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-18T01:53:10.993 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-18T01:53:10.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-18T01:53:10.996 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-09-18T01:53:15.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:15.268+0000 7f0dfa4f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4a2e800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2024-09-18T01:53:15.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:15.268+0000 7f0dfa4f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555bd4a2e800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=196, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=909fcd80-1380-4b17-b581-43d9e4cb4eca, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2024-09-18T01:53:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:Mirroring disabled 2024-09-18T01:53:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-18T01:53:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:53:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:53:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local state=deleted 2024-09-18T01:53:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:53:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local s 2024-09-18T01:53:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:53:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_10 2024-09-18T01:53:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_10 2024-09-18T01:53:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-18T01:53:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:53:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-18T01:53:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ image_id= 2024-09-18T01:53:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-18T01:53:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0.1 2024-09-18T01:53:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-18T01:53:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local test_state=deleted 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image_id= 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_state=deleted 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local current_image_id 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ get_image_id cluster1 mirror image_10 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local image=image_10 2024-09-18T01:53:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-18T01:53:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-18T01:53:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-18T01:53:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ current_image_id= 2024-09-18T01:53:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n '' 2024-09-18T01:53:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test deleted = deleted 2024-09-18T01:53:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:53:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-18T01:53:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:53:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:53:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-18T01:53:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-18T01:53:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:53:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:53:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-18T01:53:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:53:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ remove_image cluster2 mirror image_10 2024-09-18T01:53:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:53:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local image=image_10 2024-09-18T01:53:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-18T01:53:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-18T01:53:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T01:53:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:53:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-18T01:53:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ tee -a /tmp/tmp.YxZgdY10x3/rbd-mirror.test.log 2024-09-18T01:53:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ date '+%F %T' 2024-09-18T01:53:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo 2024-09-18 01:53:15 TEST: check if removed 'images'\''' OMAP are removed 2024-09-18T01:53:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:2024-09-18 01:53:15 TEST: check if removed images' OMAP are removed 2024-09-18T01:53:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-18T01:53:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:53:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-18T01:53:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:53:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local obj_name=rbd_mirroring 2024-09-18T01:53:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local filter=status_global 2024-09-18T01:53:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-18T01:53:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:53:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-18T01:53:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirroring 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local filter=status_global 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirroring 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-18T01:53:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -c status_global 2024-09-18T01:53:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 = 0 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local obj_name=rbd_mirroring 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local filter=image_ 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:53:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirroring 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local filter=image_ 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirroring 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-18T01:53:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -c image_ 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 = 0 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local obj_name=rbd_mirror_leader 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local filter=image_map 2024-09-18T01:53:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-18T01:53:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:53:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-18T01:53:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirror_leader 2024-09-18T01:53:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local filter=image_map 2024-09-18T01:53:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-18T01:53:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirror_leader 2024-09-18T01:53:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -c image_map 2024-09-18T01:53:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-18T01:53:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 1 = 0 2024-09-18T01:53:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=1 2024-09-18T01:53:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:53:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 1 -eq 0 ']' 2024-09-18T01:53:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-18T01:53:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 1 2024-09-18T01:53:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirror_leader 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local filter=image_map 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirror_leader 2024-09-18T01:53:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -c image_map 2024-09-18T01:53:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-18T01:53:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 1 = 0 2024-09-18T01:53:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=1 2024-09-18T01:53:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:53:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 1 -eq 0 ']' 2024-09-18T01:53:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-18T01:53:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 2 2024-09-18T01:53:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-18T01:53:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirror_leader 2024-09-18T01:53:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local filter=image_map 2024-09-18T01:53:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-18T01:53:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster1 2024-09-18T01:53:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -c image_map 2024-09-18T01:53:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirror_leader 2024-09-18T01:53:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-18T01:53:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 = 0 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local obj_name=rbd_mirroring 2024-09-18T01:53:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local filter=status_global 2024-09-18T01:53:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-18T01:53:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:53:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-18T01:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster2 2024-09-18T01:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirroring 2024-09-18T01:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local filter=status_global 2024-09-18T01:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-18T01:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster2 2024-09-18T01:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirroring 2024-09-18T01:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -c status_global 2024-09-18T01:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-18T01:53:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 = 0 2024-09-18T01:53:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:53:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:53:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T01:53:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:53:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-18T01:53:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:53:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local obj_name=rbd_mirroring 2024-09-18T01:53:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local filter=image_ 2024-09-18T01:53:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-18T01:53:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:53:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-18T01:53:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster2 2024-09-18T01:53:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirroring 2024-09-18T01:53:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local filter=image_ 2024-09-18T01:53:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-18T01:53:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster2 2024-09-18T01:53:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirroring 2024-09-18T01:53:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -c image_ 2024-09-18T01:53:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-18T01:53:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 = 0 2024-09-18T01:53:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:53:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local pool=mirror 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local obj_name=rbd_mirror_leader 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local filter=image_map 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ sleep 0 2024-09-18T01:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster2 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirror_leader 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local filter=image_map 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local cluster=cluster2 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local pool=mirror 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ local obj_name=rbd_mirror_leader 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ grep -c image_map 2024-09-18T01:53:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-18T01:53:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test 0 = 0 2024-09-18T01:53:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ error_code=0 2024-09-18T01:53:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set -e 2024-09-18T01:53:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T01:53:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 0 2024-09-18T01:53:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ cleanup 0 2024-09-18T01:53:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local error_code=0 2024-09-18T01:53:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ set +e 2024-09-18T01:53:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -ne 0 ']' 2024-09-18T01:53:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -z '' ']' 2024-09-18T01:53:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster instance 2024-09-18T01:53:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:53:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-18T01:53:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:pool 'mirror' does not exist 2024-09-18T01:53:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:53:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-18T01:53:21.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.147+0000 7f9487217640 -1 rbd::mirror::RemotePoolPoller: 0x5571b8fc85b0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-18T01:53:21.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.148+0000 7f9487a18640 -1 rbd::mirror::RemotePoolPoller: 0x5571b8fc85b0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-18T01:53:21.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.148+0000 7f9487a18640 -1 rbd::mirror::PoolReplayer: 0x5571b8fdbb00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-18T01:53:21.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.160+0000 7f9487217640 -1 rbd::mirror::MirrorStatusUpdater 0x5571ba1f25a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:21.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.160+0000 7f9494a32640 -1 rbd::mirror::NamespaceReplayer: 0x5571ba1f0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:21.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.161+0000 7f9487a18640 -1 rbd::mirror::MirrorStatusUpdater 0x5571ba1f2360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:21.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.161+0000 7f9494231640 -1 rbd::mirror::NamespaceReplayer: 0x5571ba1f01a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:21.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.330+0000 7f9494a32640 -1 rbd::mirror::LeaderWatcher: 0x5571ba108c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:21.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.331+0000 7f9494a32640 -1 rbd::mirror::InstanceWatcher: 0x5571ba19a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6483 object: (2) No such file or directory 2024-09-18T01:53:21.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.332+0000 7f948e225640 -1 rbd::mirror::InstanceWatcher: 0x5571ba19a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:21.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.333+0000 7f9487217640 -1 rbd::mirror::MirrorStatusUpdater 0x5571b9f506c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:21.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.333+0000 7f9494231640 -1 rbd::mirror::NamespaceReplayer: 0x5571b9dbbd40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:21.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.333+0000 7f948da24640 -1 rbd::mirror::MirrorStatusUpdater 0x5571b9ef30e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:21.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:21.333+0000 7f9494231640 -1 rbd::mirror::NamespaceReplayer: 0x5571b9dbbd40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:pool 'mirror' does not exist 2024-09-18T01:53:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:53:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-18T01:53:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:pool 'mirror_parent' does not exist 2024-09-18T01:53:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ CEPH_ARGS= 2024-09-18T01:53:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-18T01:53:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.948+0000 7f0e06d0f640 -1 rbd::mirror::LeaderWatcher: 0x555bd450ec00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-18T01:53:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0e00502640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555bd49f0900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-18T01:53:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0e00502640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555bd4501440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-18T01:53:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0e00502640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555bd4500480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-18T01:53:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0e00502640 -1 rbd::mirror::PoolWatcher: 0x555bd4c17a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0e00502640 -1 rbd::mirror::PoolWatcher: 0x555bd4b4e960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0df9cf5640 -1 rbd::mirror::PoolWatcher: 0x555bd4b4e000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:22.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0df9cf5640 -1 rbd::mirror::PoolWatcher: 0x555bd4acc5a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:22.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0e00d03640 -1 rbd::mirror::PoolWatcher: 0x555bd4b4eb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:22.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0df9cf5640 -1 rbd::mirror::PoolWatcher: 0x555bd4b4f860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:22.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0e06d0f640 -1 rbd::mirror::LeaderWatcher: 0x555bd450ec00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-18T01:53:22.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:22.949+0000 7f0e06d0f640 -1 rbd::mirror::LeaderWatcher: 0x555bd450ec00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-18T01:53:23.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.113+0000 7f2152d00640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56083efc2000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-18T01:53:23.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.113+0000 7f2152d00640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56083e9ca480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-18T01:53:23.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.113+0000 7f2153501640 -1 rbd::mirror::PoolWatcher: 0x56083efb5e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:23.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.113+0000 7f2153501640 -1 rbd::mirror::PoolWatcher: 0x56083efb54a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:23.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.115+0000 7f214c4f3640 -1 rbd::mirror::PoolWatcher: 0x56083f1e45a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:23.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.115+0000 7f214c4f3640 -1 rbd::mirror::PoolWatcher: 0x56083efb4b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:23.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.115+0000 7f215950d640 -1 rbd::mirror::InstanceWatcher: 0x56083ea72380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4402 object: (2) No such file or directory 2024-09-18T01:53:23.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.115+0000 7f215950d640 -1 rbd::mirror::InstanceWatcher: 0x56083ea72700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4402 object: (2) No such file or directory 2024-09-18T01:53:23.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f2153501640 -1 rbd::mirror::InstanceWatcher: 0x56083ea72380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f2153501640 -1 rbd::mirror::InstanceWatcher: 0x56083ea72700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f214ccf4640 -1 rbd::mirror::MirrorStatusUpdater 0x56083eab6360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f214ccf4640 -1 rbd::mirror::MirrorStatusUpdater 0x56083eab65a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f215950d640 -1 rbd::mirror::NamespaceReplayer: 0x56083eab2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f215950d640 -1 rbd::mirror::NamespaceReplayer: 0x56083eab21a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f2152d00640 -1 rbd::mirror::MirrorStatusUpdater 0x56083eab6000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f2152d00640 -1 rbd::mirror::MirrorStatusUpdater 0x56083eab6120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f2159d0e640 -1 rbd::mirror::NamespaceReplayer: 0x56083eab2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.116+0000 7f2159d0e640 -1 rbd::mirror::NamespaceReplayer: 0x56083eab21a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.162+0000 7f214ccf4640 -1 rbd::mirror::RemotePoolPoller: 0x56083d8b0ea0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-18T01:53:23.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.162+0000 7f214c4f3640 -1 rbd::mirror::RemotePoolPoller: 0x56083d8b0ea0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-18T01:53:23.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.162+0000 7f214c4f3640 -1 rbd::mirror::PoolReplayer: 0x56083d8c3b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-18T01:53:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:pool 'mirror_parent' does not exist 2024-09-18T01:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-18T01:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirrors cluster1 2024-09-18T01:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1 2024-09-18T01:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 0 3 2024-09-18T01:53:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:53:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirror cluster1:0 '' 2024-09-18T01:53:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1:0 2024-09-18T01:53:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:53:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:53:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirror cluster1:1 '' 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1:1 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirror cluster1:2 '' 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1:2 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirror cluster1:3 '' 2024-09-18T01:53:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster1:3 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirrors cluster2 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:++ seq 0 3 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirror cluster2:0 '' 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2:0 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:53:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirror cluster2:1 '' 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2:1 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirror cluster2:2 '' 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2:2 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T01:53:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ stop_mirror cluster2:3 '' 2024-09-18T01:53:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local cluster=cluster2:3 2024-09-18T01:53:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ local sig= 2024-09-18T01:53:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test -n 1 2024-09-18T01:53:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ return 2024-09-18T01:53:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' -z 1 ']' 2024-09-18T01:53:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ test '' = /tmp/tmp.YxZgdY10x3 2024-09-18T01:53:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ rm -Rf /tmp/tmp.YxZgdY10x3 2024-09-18T01:53:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T01:53:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ echo OK 2024-09-18T01:53:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stdout:OK 2024-09-18T01:53:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi115.stderr:+ exit 0 2024-09-18T01:53:23.399 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T01:53:23.399 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-18T01:53:23.488 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-18T01:53:23.488 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-18T01:53:23.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.582+0000 7f2153501640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56083e9cad80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-18T01:53:23.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.582+0000 7f2152d00640 -1 rbd::mirror::PoolWatcher: 0x56083f04da40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:23.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.582+0000 7f214ccf4640 -1 rbd::mirror::PoolWatcher: 0x56083f1e4960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.582+0000 7f2159d0e640 -1 rbd::mirror::LeaderWatcher: 0x56083e9e0c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-18T01:53:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.582+0000 7f2159d0e640 -1 rbd::mirror::LeaderWatcher: 0x56083e9e0c00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-18T01:53:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.582+0000 7f215950d640 -1 rbd::mirror::LeaderWatcher: 0x56083e9e0c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.583+0000 7f215950d640 -1 rbd::mirror::InstanceWatcher: 0x56083ea72000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4402 object: (2) No such file or directory 2024-09-18T01:53:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.583+0000 7f2152d00640 -1 rbd::mirror::InstanceWatcher: 0x56083ea72000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.583+0000 7f214c4f3640 -1 rbd::mirror::MirrorStatusUpdater 0x56083e82a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.583+0000 7f2159d0e640 -1 rbd::mirror::NamespaceReplayer: 0x56083f2d2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.584+0000 7f2153501640 -1 rbd::mirror::MirrorStatusUpdater 0x56083e7d30e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.584+0000 7f215950d640 -1 rbd::mirror::NamespaceReplayer: 0x56083f2d2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.586+0000 7f2159d0e640 -1 rbd::mirror::LeaderWatcher: 0x56083ec73b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:23.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.586+0000 7f21454e5640 -1 rbd::mirror::InstanceWatcher: 0x56083ef3a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.587+0000 7f213f4d9640 -1 rbd::mirror::MirrorStatusUpdater 0x56083eab66c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.587+0000 7f215950d640 -1 rbd::mirror::NamespaceReplayer: 0x56083e6a5a00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.587+0000 7f2145ce6640 -1 rbd::mirror::MirrorStatusUpdater 0x56083ecb9680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.587+0000 7f2159d0e640 -1 rbd::mirror::NamespaceReplayer: 0x56083e6a5a00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.737 DEBUG:teuthology.parallel:result is None 2024-09-18T01:53:23.737 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-18T01:53:23.762 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-18T01:53:23.763 DEBUG:teuthology.orchestra.run.smithi115:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-18T01:53:23.828 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-18T01:53:23.829 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-18T01:53:23.838 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-18T01:53:23.838 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 14 2024-09-18T01:53:23.839 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-18T01:53:23.839 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-18T01:53:23.860 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-18T01:53:23.860 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 12 2024-09-18T01:53:23.860 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-18T01:53:23.860 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-18T01:53:23.871 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-18T01:53:23.871 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.831+0000 7f0e06d0f640 -1 rbd::mirror::LeaderWatcher: 0x555bd4946300 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-18T01:53:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.832+0000 7f0df34e8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555bd4500d80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-18T01:53:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.832+0000 7f0deccdb640 -1 rbd::mirror::PoolWatcher: 0x555bd4d041e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.833+0000 7f0e06d0f640 -1 rbd::mirror::LeaderWatcher: 0x555bd4946300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-18T01:53:23.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.833+0000 7f0e06d0f640 -1 rbd::mirror::LeaderWatcher: 0x555bd4946300 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-18T01:53:23.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.871+0000 7f9495233640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 68870) UID: 1000 2024-09-18T01:53:23.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.871+0000 7f9494231640 -1 rbd::mirror::LeaderWatcher: 0x5571ba3b9200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:23.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.871+0000 7f9494231640 -1 rbd::mirror::InstanceWatcher: 0x5571ba19a700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6485 object: (2) No such file or directory 2024-09-18T01:53:23.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.872+0000 7f9480209640 -1 rbd::mirror::InstanceWatcher: 0x5571ba19a700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.872+0000 7f947a1fd640 -1 rbd::mirror::MirrorStatusUpdater 0x5571ba3ee7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.872+0000 7f9494231640 -1 rbd::mirror::NamespaceReplayer: 0x5571ba66e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.872+0000 7f9480a0a640 -1 rbd::mirror::MirrorStatusUpdater 0x5571ba3e7200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi115.stderr:2024-09-18T01:53:23.872+0000 7f9494a32640 -1 rbd::mirror::NamespaceReplayer: 0x5571ba66e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-18T01:53:23.884 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-18T01:53:23.894 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-18T01:53:23.895 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:23.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi115.stderr:2024-09-18T01:53:23.925+0000 7f215a50f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 40036) UID: 1000 2024-09-18T01:53:23.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-18T01:53:23.936 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-18T01:53:23.946 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-18T01:53:23.947 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:23.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.976+0000 7f4b76ea2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 71920) UID: 1000 2024-09-18T01:53:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.976+0000 7f4b75ea0640 -1 rbd::mirror::LeaderWatcher: 0x5573143a5200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b75ea0640 -1 rbd::mirror::InstanceWatcher: 0x557314196380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6738 object: (2) No such file or directory 2024-09-18T01:53:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b75ea0640 -1 rbd::mirror::InstanceWatcher: 0x557314682000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6738 object: (2) No such file or directory 2024-09-18T01:53:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b75ea0640 -1 rbd::mirror::InstanceWatcher: 0x557314197c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6741 object: (2) No such file or directory 2024-09-18T01:53:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b6fe94640 -1 rbd::mirror::InstanceWatcher: 0x557314196380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b6fe94640 -1 rbd::mirror::InstanceWatcher: 0x557314682000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b61e78640 -1 rbd::mirror::InstanceWatcher: 0x557314197c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b6f693640 -1 rbd::mirror::MirrorStatusUpdater 0x5573141ec000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b6f693640 -1 rbd::mirror::MirrorStatusUpdater 0x5573141ec120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b75ea0640 -1 rbd::mirror::NamespaceReplayer: 0x5573141ea000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.977+0000 7f4b75ea0640 -1 rbd::mirror::NamespaceReplayer: 0x5573141ea1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.979+0000 7f4b62679640 -1 rbd::mirror::MirrorStatusUpdater 0x5573143c0fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.979+0000 7f4b75ea0640 -1 rbd::mirror::NamespaceReplayer: 0x5573141ea4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.981+0000 7f4b75ea0640 -1 rbd::mirror::LeaderWatcher: 0x557314102c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:23.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.981+0000 7f4b75ea0640 -1 rbd::mirror::InstanceWatcher: 0x557314196000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6738 object: (2) No such file or directory 2024-09-18T01:53:23.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.982+0000 7f4b6fe94640 -1 rbd::mirror::InstanceWatcher: 0x557314196000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:23.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.982+0000 7f4b6f693640 -1 rbd::mirror::MirrorStatusUpdater 0x557313efd0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:23.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi115.stderr:2024-09-18T01:53:23.982+0000 7f4b766a1640 -1 rbd::mirror::NamespaceReplayer: 0x557313dc81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:23.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-18T01:53:23.993 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-18T01:53:24.004 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-18T01:53:24.004 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:24.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.034+0000 7fb754bae640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 71110) UID: 1000 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.034+0000 7fb73fb84640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55feb0c64480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb7543ad640 -1 rbd::mirror::InstanceWatcher: 0x55feb07fa380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6671 object: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb740385640 -1 rbd::mirror::PoolWatcher: 0x55feb0d61c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb7543ad640 -1 rbd::mirror::InstanceWatcher: 0x55feb0d02000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6671 object: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb74d39f640 -1 rbd::mirror::InstanceWatcher: 0x55feb07fa380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb74dba0640 -1 rbd::mirror::InstanceWatcher: 0x55feb0d02000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb753bac640 -1 rbd::mirror::LeaderWatcher: 0x55feb0a21200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb753bac640 -1 rbd::mirror::LeaderWatcher: 0x55feb0a21200 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb7543ad640 -1 rbd::mirror::LeaderWatcher: 0x55feb0a21200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb746b92640 -1 rbd::mirror::MirrorStatusUpdater 0x55feb0870360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb746b92640 -1 rbd::mirror::MirrorStatusUpdater 0x55feb0a54c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb753bac640 -1 rbd::mirror::NamespaceReplayer: 0x55feb086c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.035+0000 7fb753bac640 -1 rbd::mirror::NamespaceReplayer: 0x55feb086c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.036+0000 7fb7543ad640 -1 rbd::mirror::InstanceWatcher: 0x55feb07fa700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6674 object: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.036+0000 7fb74dba0640 -1 rbd::mirror::MirrorStatusUpdater 0x55feb0870000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.036+0000 7fb74dba0640 -1 rbd::mirror::MirrorStatusUpdater 0x55feb0870120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.036+0000 7fb753bac640 -1 rbd::mirror::NamespaceReplayer: 0x55feb086c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.036+0000 7fb753bac640 -1 rbd::mirror::NamespaceReplayer: 0x55feb086c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.036+0000 7fb740385640 -1 rbd::mirror::InstanceWatcher: 0x55feb07fa700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.036+0000 7fb73fb84640 -1 rbd::mirror::MirrorStatusUpdater 0x55feb0a0cd80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.036+0000 7fb7543ad640 -1 rbd::mirror::NamespaceReplayer: 0x55feb086c4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.038+0000 7fb7543ad640 -1 rbd::mirror::LeaderWatcher: 0x55feb0768c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.038+0000 7fb753bac640 -1 rbd::mirror::InstanceWatcher: 0x55feb07fa000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6671 object: (2) No such file or directory 2024-09-18T01:53:24.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.038+0000 7fb74d39f640 -1 rbd::mirror::InstanceWatcher: 0x55feb07fa000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.039+0000 7fb747393640 -1 rbd::mirror::MirrorStatusUpdater 0x55feb05b46c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.039+0000 7fb7543ad640 -1 rbd::mirror::NamespaceReplayer: 0x55feb0474340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.039+0000 7fb74dba0640 -1 rbd::mirror::MirrorStatusUpdater 0x55feb05370e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.039+0000 7fb753bac640 -1 rbd::mirror::NamespaceReplayer: 0x55feb0474340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-18T01:53:24.050 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-18T01:53:24.061 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-18T01:53:24.061 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:24.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.091+0000 7f0e07d11640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 61629) UID: 1000 2024-09-18T01:53:24.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.092+0000 7f0e06d0f640 -1 rbd::mirror::LeaderWatcher: 0x555bd4946300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.092+0000 7f0e07510640 -1 rbd::mirror::InstanceWatcher: 0x555bd45b0380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5606 object: (2) No such file or directory 2024-09-18T01:53:24.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.092+0000 7f0e06d0f640 -1 rbd::mirror::InstanceWatcher: 0x555bd45b0700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5606 object: (2) No such file or directory 2024-09-18T01:53:24.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.092+0000 7f0e07510640 -1 rbd::mirror::InstanceWatcher: 0x555bd4a92000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5609 object: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.092+0000 7f0e00502640 -1 rbd::mirror::InstanceWatcher: 0x555bd45b0380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.092+0000 7f0e00d03640 -1 rbd::mirror::InstanceWatcher: 0x555bd45b0700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0df34e8640 -1 rbd::mirror::InstanceWatcher: 0x555bd4a92000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0dfa4f6640 -1 rbd::mirror::MirrorStatusUpdater 0x555bd4624360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0dfa4f6640 -1 rbd::mirror::MirrorStatusUpdater 0x555bd4624480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0e07510640 -1 rbd::mirror::NamespaceReplayer: 0x555bd4622000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0e07510640 -1 rbd::mirror::NamespaceReplayer: 0x555bd46221a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0deccdb640 -1 rbd::mirror::MirrorStatusUpdater 0x555bd4624b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0e06d0f640 -1 rbd::mirror::NamespaceReplayer: 0x555bd4a8a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0e00502640 -1 rbd::mirror::MirrorStatusUpdater 0x555bd4624000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0e07510640 -1 rbd::mirror::NamespaceReplayer: 0x555bd4622000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0e00d03640 -1 rbd::mirror::MirrorStatusUpdater 0x555bd4624120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0e06d0f640 -1 rbd::mirror::NamespaceReplayer: 0x555bd46221a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0df2ce7640 -1 rbd::mirror::MirrorStatusUpdater 0x555bd47d9440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.093+0000 7f0e07510640 -1 rbd::mirror::NamespaceReplayer: 0x555bd4a8a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.095+0000 7f0e07510640 -1 rbd::mirror::LeaderWatcher: 0x555bd450ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.096+0000 7f0e07510640 -1 rbd::mirror::InstanceWatcher: 0x555bd45b0000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5606 object: (2) No such file or directory 2024-09-18T01:53:24.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.096+0000 7f0e00502640 -1 rbd::mirror::InstanceWatcher: 0x555bd45b0000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.096+0000 7f0df9cf5640 -1 rbd::mirror::MirrorStatusUpdater 0x555bd43666c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.096+0000 7f0e06d0f640 -1 rbd::mirror::NamespaceReplayer: 0x555bd340e680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.096+0000 7f0e00d03640 -1 rbd::mirror::MirrorStatusUpdater 0x555bd42f4fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi115.stderr:2024-09-18T01:53:24.096+0000 7f0e07510640 -1 rbd::mirror::NamespaceReplayer: 0x555bd340e680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-18T01:53:24.109 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-18T01:53:24.120 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-18T01:53:24.120 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:24.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.150+0000 7ff050c7b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 45725) UID: 1000 2024-09-18T01:53:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.150+0000 7ff04fc79640 -1 rbd::mirror::LeaderWatcher: 0x55ee1312d200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff05047a640 -1 rbd::mirror::InstanceWatcher: 0x55ee12ee4380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4713 object: (2) No such file or directory 2024-09-18T01:53:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff05047a640 -1 rbd::mirror::InstanceWatcher: 0x55ee133c2000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4713 object: (2) No such file or directory 2024-09-18T01:53:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff05047a640 -1 rbd::mirror::InstanceWatcher: 0x55ee12ee5c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4716 object: (2) No such file or directory 2024-09-18T01:53:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff04946c640 -1 rbd::mirror::InstanceWatcher: 0x55ee12ee4380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff04946c640 -1 rbd::mirror::InstanceWatcher: 0x55ee133c2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff03bc51640 -1 rbd::mirror::InstanceWatcher: 0x55ee12ee5c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff042c5f640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee12f52360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff042c5f640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee13154a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff05047a640 -1 rbd::mirror::NamespaceReplayer: 0x55ee12eda340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff05047a640 -1 rbd::mirror::NamespaceReplayer: 0x55ee12eda4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff035444640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee13154480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff04fc79640 -1 rbd::mirror::NamespaceReplayer: 0x55ee12eda820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff049c6d640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee12f52000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff049c6d640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee12f52120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff05047a640 -1 rbd::mirror::NamespaceReplayer: 0x55ee12eda340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.151+0000 7ff05047a640 -1 rbd::mirror::NamespaceReplayer: 0x55ee12eda4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.152+0000 7ff03c452640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee130e4c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.152+0000 7ff04fc79640 -1 rbd::mirror::NamespaceReplayer: 0x55ee12eda820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.154+0000 7ff05047a640 -1 rbd::mirror::LeaderWatcher: 0x55ee12e36c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.154+0000 7ff04fc79640 -1 rbd::mirror::InstanceWatcher: 0x55ee12ee4000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4713 object: (2) No such file or directory 2024-09-18T01:53:24.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.155+0000 7ff04946c640 -1 rbd::mirror::InstanceWatcher: 0x55ee12ee4000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.155+0000 7ff043460640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee12bed0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.155+0000 7ff04fc79640 -1 rbd::mirror::NamespaceReplayer: 0x55ee12eda000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.155+0000 7ff049c6d640 -1 rbd::mirror::MirrorStatusUpdater 0x55ee12bed680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi115.stderr:2024-09-18T01:53:24.155+0000 7ff05047a640 -1 rbd::mirror::NamespaceReplayer: 0x55ee12eda000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-18T01:53:24.167 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-18T01:53:24.179 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-18T01:53:24.179 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:24.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.208+0000 7f747f8f0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 71559) UID: 1000 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f747e8ee640 -1 rbd::mirror::InstanceWatcher: 0x56233e5c4380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6146 object: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f747f0ef640 -1 rbd::mirror::InstanceWatcher: 0x56233e5c4a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6146 object: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f746a8c6640 -1 rbd::mirror::InstanceWatcher: 0x56233ea7c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f74780e1640 -1 rbd::mirror::InstanceWatcher: 0x56233e5c4380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f74780e1640 -1 rbd::mirror::InstanceWatcher: 0x56233e5c4a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f74640b9640 -1 rbd::mirror::MirrorStatusUpdater 0x56233e6326c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f747e8ee640 -1 rbd::mirror::NamespaceReplayer: 0x56233e6304e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f74720d5640 -1 rbd::mirror::MirrorStatusUpdater 0x56233e632480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f74720d5640 -1 rbd::mirror::MirrorStatusUpdater 0x56233e632360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f747f0ef640 -1 rbd::mirror::NamespaceReplayer: 0x56233e630000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.209+0000 7f747f0ef640 -1 rbd::mirror::NamespaceReplayer: 0x56233e6301a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.210+0000 7f74788e2640 -1 rbd::mirror::MirrorStatusUpdater 0x56233e632000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.210+0000 7f747e8ee640 -1 rbd::mirror::NamespaceReplayer: 0x56233e630000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.210+0000 7f74780e1640 -1 rbd::mirror::MirrorStatusUpdater 0x56233e632120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.210+0000 7f747f0ef640 -1 rbd::mirror::NamespaceReplayer: 0x56233e6301a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.211+0000 7f747f0ef640 -1 rbd::mirror::LeaderWatcher: 0x56233e52cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.212+0000 7f747f0ef640 -1 rbd::mirror::InstanceWatcher: 0x56233e5c4000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6146 object: (2) No such file or directory 2024-09-18T01:53:24.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.212+0000 7f74788e2640 -1 rbd::mirror::InstanceWatcher: 0x56233e5c4000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.212+0000 7f74718d4640 -1 rbd::mirror::MirrorStatusUpdater 0x56233e3826c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.212+0000 7f747f0ef640 -1 rbd::mirror::NamespaceReplayer: 0x56233ee42000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.212+0000 7f74780e1640 -1 rbd::mirror::MirrorStatusUpdater 0x56233e320fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi115.stderr:2024-09-18T01:53:24.212+0000 7f747e8ee640 -1 rbd::mirror::NamespaceReplayer: 0x56233ee42000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-18T01:53:24.224 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-18T01:53:24.235 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-18T01:53:24.236 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:24.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.264+0000 7f85cbb06640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 70190) UID: 1000 2024-09-18T01:53:24.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85cb305640 -1 rbd::mirror::LeaderWatcher: 0x563c2ce89200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85cab04640 -1 rbd::mirror::InstanceWatcher: 0x563c2cc20380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6077 object: (2) No such file or directory 2024-09-18T01:53:24.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85cab04640 -1 rbd::mirror::InstanceWatcher: 0x563c2d14c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6077 object: (2) No such file or directory 2024-09-18T01:53:24.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85cab04640 -1 rbd::mirror::InstanceWatcher: 0x563c2cc21c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6079 object: (2) No such file or directory 2024-09-18T01:53:24.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85c4af8640 -1 rbd::mirror::InstanceWatcher: 0x563c2cc20380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85c42f7640 -1 rbd::mirror::InstanceWatcher: 0x563c2d14c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85be2eb640 -1 rbd::mirror::MirrorStatusUpdater 0x563c2cc6c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85be2eb640 -1 rbd::mirror::MirrorStatusUpdater 0x563c2cc6c7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85cb305640 -1 rbd::mirror::NamespaceReplayer: 0x563c2cc9c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.265+0000 7f85cb305640 -1 rbd::mirror::NamespaceReplayer: 0x563c2cc9c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85b72dd640 -1 rbd::mirror::InstanceWatcher: 0x563c2cc21c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85c4af8640 -1 rbd::mirror::MirrorStatusUpdater 0x563c2cc6c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85c4af8640 -1 rbd::mirror::MirrorStatusUpdater 0x563c2cc6c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85cb305640 -1 rbd::mirror::NamespaceReplayer: 0x563c2cc9c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85cb305640 -1 rbd::mirror::NamespaceReplayer: 0x563c2cc9c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85b0ad0640 -1 rbd::mirror::MirrorStatusUpdater 0x563c2cebe7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85cab04640 -1 rbd::mirror::NamespaceReplayer: 0x563c2cfd41a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85b6adc640 -1 rbd::mirror::MirrorStatusUpdater 0x563c2ce5f0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.266+0000 7f85cb305640 -1 rbd::mirror::NamespaceReplayer: 0x563c2cfd41a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.268+0000 7f85cb305640 -1 rbd::mirror::LeaderWatcher: 0x563c2cb80c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-18T01:53:24.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.269+0000 7f85cb305640 -1 rbd::mirror::InstanceWatcher: 0x563c2cc20000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6077 object: (2) No such file or directory 2024-09-18T01:53:24.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.269+0000 7f85c42f7640 -1 rbd::mirror::InstanceWatcher: 0x563c2cc20000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-18T01:53:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.270+0000 7f85bdaea640 -1 rbd::mirror::MirrorStatusUpdater 0x563c2c9dc6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.270+0000 7f85cab04640 -1 rbd::mirror::NamespaceReplayer: 0x563c2ba82680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.270+0000 7f85c4af8640 -1 rbd::mirror::MirrorStatusUpdater 0x563c2c96efc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-18T01:53:24.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi115.stderr:2024-09-18T01:53:24.270+0000 7f85cb305640 -1 rbd::mirror::NamespaceReplayer: 0x563c2ba82680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-18T01:53:24.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-18T01:53:24.282 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T01:53:24.292 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-18T01:53:24.292 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T01:53:24.637 INFO:teuthology.orchestra.run.smithi115.stderr:2024-09-18T01:53:24.637+0000 7f5dfa7fc640 0 --2- 172.21.15.115:0/989366528 >> v2:172.21.15.115:3300/0 conn(0x7f5dfc1516f0 0x7f5dfc0af230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T01:53:24.811 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:53:24.811 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-18T01:53:24.823 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-09-18T01:53:23.975925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4255488,"kb_used_data":17728,"kb_used_omap":4,"kb_used_meta":4237691,"kb_avail":278860032,"statfs":{"total":289910292480,"available":285552672768,"internally_reserved":0,"allocated":18153472,"data_stored":10171149,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4339395919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536832238,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-138025,"num_read_kb":-39828039,"num_write":-599093,"num_write_kb":-219651984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":-8,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002115"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.448253+0000","last_change":"2024-09-18T01:17:02.518768+0000","last_active":"2024-09-18T01:53:23.448253+0000","last_peered":"2024-09-18T01:53:23.448253+0000","last_clean":"2024-09-18T01:53:23.448253+0000","last_became_active":"2024-09-18T01:17:00.511441+0000","last_became_peered":"2024-09-18T01:17:00.511441+0000","last_unstale":"2024-09-18T01:53:23.448253+0000","last_undegraded":"2024-09-18T01:53:23.448253+0000","last_fullsized":"2024-09-18T01:53:23.448253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:23:40.338890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000687489,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.455003+0000","last_change":"2024-09-18T01:17:02.518477+0000","last_active":"2024-09-18T01:53:23.455003+0000","last_peered":"2024-09-18T01:53:23.455003+0000","last_clean":"2024-09-18T01:53:23.455003+0000","last_became_active":"2024-09-18T01:17:00.510105+0000","last_became_peered":"2024-09-18T01:17:00.510105+0000","last_unstale":"2024-09-18T01:53:23.455003+0000","last_undegraded":"2024-09-18T01:53:23.455003+0000","last_fullsized":"2024-09-18T01:53:23.455003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:12:46.477909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043520000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.445597+0000","last_change":"2024-09-18T01:17:02.519127+0000","last_active":"2024-09-18T01:53:23.445597+0000","last_peered":"2024-09-18T01:53:23.445597+0000","last_clean":"2024-09-18T01:53:23.445597+0000","last_became_active":"2024-09-18T01:17:00.509844+0000","last_became_peered":"2024-09-18T01:17:00.509844+0000","last_unstale":"2024-09-18T01:53:23.445597+0000","last_undegraded":"2024-09-18T01:53:23.445597+0000","last_fullsized":"2024-09-18T01:53:23.445597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:45:37.121984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.450865+0000","last_change":"2024-09-18T01:17:02.518480+0000","last_active":"2024-09-18T01:53:23.450865+0000","last_peered":"2024-09-18T01:53:23.450865+0000","last_clean":"2024-09-18T01:53:23.450865+0000","last_became_active":"2024-09-18T01:17:00.509856+0000","last_became_peered":"2024-09-18T01:17:00.509856+0000","last_unstale":"2024-09-18T01:53:23.450865+0000","last_undegraded":"2024-09-18T01:53:23.450865+0000","last_fullsized":"2024-09-18T01:53:23.450865+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:34:28.368843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042899300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1020,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.447312+0000","last_change":"2024-09-18T01:17:02.513682+0000","last_active":"2024-09-18T01:53:23.447312+0000","last_peered":"2024-09-18T01:53:23.447312+0000","last_clean":"2024-09-18T01:53:23.447312+0000","last_became_active":"2024-09-18T01:17:00.511011+0000","last_became_peered":"2024-09-18T01:17:00.511011+0000","last_unstale":"2024-09-18T01:53:23.447312+0000","last_undegraded":"2024-09-18T01:53:23.447312+0000","last_fullsized":"2024-09-18T01:53:23.447312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:14:05.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011029220000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1011,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-18T01:53:15.253798+0000","last_change":"2024-09-18T01:17:02.511032+0000","last_active":"2024-09-18T01:53:15.253798+0000","last_peered":"2024-09-18T01:53:15.253798+0000","last_clean":"2024-09-18T01:53:15.253798+0000","last_became_active":"2024-09-18T01:17:00.510439+0000","last_became_peered":"2024-09-18T01:17:00.510439+0000","last_unstale":"2024-09-18T01:53:15.253798+0000","last_undegraded":"2024-09-18T01:53:15.253798+0000","last_fullsized":"2024-09-18T01:53:15.253798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:04:11.663327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1012,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-18T01:53:15.260633+0000","last_change":"2024-09-18T01:17:02.510963+0000","last_active":"2024-09-18T01:53:15.260633+0000","last_peered":"2024-09-18T01:53:15.260633+0000","last_clean":"2024-09-18T01:53:15.260633+0000","last_became_active":"2024-09-18T01:17:00.512356+0000","last_became_peered":"2024-09-18T01:17:00.512356+0000","last_unstale":"2024-09-18T01:53:15.260633+0000","last_undegraded":"2024-09-18T01:53:15.260633+0000","last_fullsized":"2024-09-18T01:53:15.260633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:42:49.393044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1019,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.444116+0000","last_change":"2024-09-18T01:17:02.518778+0000","last_active":"2024-09-18T01:53:23.444116+0000","last_peered":"2024-09-18T01:53:23.444116+0000","last_clean":"2024-09-18T01:53:23.444116+0000","last_became_active":"2024-09-18T01:17:00.509327+0000","last_became_peered":"2024-09-18T01:17:00.509327+0000","last_unstale":"2024-09-18T01:53:23.444116+0000","last_undegraded":"2024-09-18T01:53:23.444116+0000","last_fullsized":"2024-09-18T01:53:23.444116+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:26:58.168836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069378899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1147,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.443198+0000","last_change":"2024-09-18T01:16:55.498135+0000","last_active":"2024-09-18T01:53:23.443198+0000","last_peered":"2024-09-18T01:53:23.443198+0000","last_clean":"2024-09-18T01:53:23.443198+0000","last_became_active":"2024-09-18T01:16:55.497021+0000","last_became_peered":"2024-09-18T01:16:55.497021+0000","last_unstale":"2024-09-18T01:53:23.443198+0000","last_undegraded":"2024-09-18T01:53:23.443198+0000","last_fullsized":"2024-09-18T01:53:23.443198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_clean_scrub_stamp":"2024-09-18T01:16:54.479487+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:24:07.471201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1287352,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":1281470,"kb_avail":93084488,"statfs":{"total":96636764160,"available":95318515712,"internally_reserved":0,"allocated":5996544,"data_stored":2989833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1312225733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706105,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1544892,"kb_used_data":5928,"kb_used_omap":1,"kb_used_meta":1538942,"kb_avail":92826948,"statfs":{"total":96636764160,"available":95054794752,"internally_reserved":0,"allocated":6070272,"data_stored":3587324,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1575877061},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1423244,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":1417278,"kb_avail":92948596,"statfs":{"total":96636764160,"available":95179362304,"internally_reserved":0,"allocated":6086656,"data_stored":3593992,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1451293125},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:24.824 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T01:53:25.097 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:53:25.097 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-18T01:53:25.109 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-09-18T01:53:23.975925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4255488,"kb_used_data":17728,"kb_used_omap":4,"kb_used_meta":4237691,"kb_avail":278860032,"statfs":{"total":289910292480,"available":285552672768,"internally_reserved":0,"allocated":18153472,"data_stored":10171149,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4339395919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536832238,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-138025,"num_read_kb":-39828039,"num_write":-599093,"num_write_kb":-219651984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":-8,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002115"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.448253+0000","last_change":"2024-09-18T01:17:02.518768+0000","last_active":"2024-09-18T01:53:23.448253+0000","last_peered":"2024-09-18T01:53:23.448253+0000","last_clean":"2024-09-18T01:53:23.448253+0000","last_became_active":"2024-09-18T01:17:00.511441+0000","last_became_peered":"2024-09-18T01:17:00.511441+0000","last_unstale":"2024-09-18T01:53:23.448253+0000","last_undegraded":"2024-09-18T01:53:23.448253+0000","last_fullsized":"2024-09-18T01:53:23.448253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:23:40.338890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000687489,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.455003+0000","last_change":"2024-09-18T01:17:02.518477+0000","last_active":"2024-09-18T01:53:23.455003+0000","last_peered":"2024-09-18T01:53:23.455003+0000","last_clean":"2024-09-18T01:53:23.455003+0000","last_became_active":"2024-09-18T01:17:00.510105+0000","last_became_peered":"2024-09-18T01:17:00.510105+0000","last_unstale":"2024-09-18T01:53:23.455003+0000","last_undegraded":"2024-09-18T01:53:23.455003+0000","last_fullsized":"2024-09-18T01:53:23.455003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:12:46.477909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043520000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.445597+0000","last_change":"2024-09-18T01:17:02.519127+0000","last_active":"2024-09-18T01:53:23.445597+0000","last_peered":"2024-09-18T01:53:23.445597+0000","last_clean":"2024-09-18T01:53:23.445597+0000","last_became_active":"2024-09-18T01:17:00.509844+0000","last_became_peered":"2024-09-18T01:17:00.509844+0000","last_unstale":"2024-09-18T01:53:23.445597+0000","last_undegraded":"2024-09-18T01:53:23.445597+0000","last_fullsized":"2024-09-18T01:53:23.445597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:45:37.121984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.450865+0000","last_change":"2024-09-18T01:17:02.518480+0000","last_active":"2024-09-18T01:53:23.450865+0000","last_peered":"2024-09-18T01:53:23.450865+0000","last_clean":"2024-09-18T01:53:23.450865+0000","last_became_active":"2024-09-18T01:17:00.509856+0000","last_became_peered":"2024-09-18T01:17:00.509856+0000","last_unstale":"2024-09-18T01:53:23.450865+0000","last_undegraded":"2024-09-18T01:53:23.450865+0000","last_fullsized":"2024-09-18T01:53:23.450865+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:34:28.368843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042899300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1020,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.447312+0000","last_change":"2024-09-18T01:17:02.513682+0000","last_active":"2024-09-18T01:53:23.447312+0000","last_peered":"2024-09-18T01:53:23.447312+0000","last_clean":"2024-09-18T01:53:23.447312+0000","last_became_active":"2024-09-18T01:17:00.511011+0000","last_became_peered":"2024-09-18T01:17:00.511011+0000","last_unstale":"2024-09-18T01:53:23.447312+0000","last_undegraded":"2024-09-18T01:53:23.447312+0000","last_fullsized":"2024-09-18T01:53:23.447312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:14:05.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011029220000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1011,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-18T01:53:15.253798+0000","last_change":"2024-09-18T01:17:02.511032+0000","last_active":"2024-09-18T01:53:15.253798+0000","last_peered":"2024-09-18T01:53:15.253798+0000","last_clean":"2024-09-18T01:53:15.253798+0000","last_became_active":"2024-09-18T01:17:00.510439+0000","last_became_peered":"2024-09-18T01:17:00.510439+0000","last_unstale":"2024-09-18T01:53:15.253798+0000","last_undegraded":"2024-09-18T01:53:15.253798+0000","last_fullsized":"2024-09-18T01:53:15.253798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:04:11.663327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1012,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-18T01:53:15.260633+0000","last_change":"2024-09-18T01:17:02.510963+0000","last_active":"2024-09-18T01:53:15.260633+0000","last_peered":"2024-09-18T01:53:15.260633+0000","last_clean":"2024-09-18T01:53:15.260633+0000","last_became_active":"2024-09-18T01:17:00.512356+0000","last_became_peered":"2024-09-18T01:17:00.512356+0000","last_unstale":"2024-09-18T01:53:15.260633+0000","last_undegraded":"2024-09-18T01:53:15.260633+0000","last_fullsized":"2024-09-18T01:53:15.260633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:42:49.393044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1019,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.444116+0000","last_change":"2024-09-18T01:17:02.518778+0000","last_active":"2024-09-18T01:53:23.444116+0000","last_peered":"2024-09-18T01:53:23.444116+0000","last_clean":"2024-09-18T01:53:23.444116+0000","last_became_active":"2024-09-18T01:17:00.509327+0000","last_became_peered":"2024-09-18T01:17:00.509327+0000","last_unstale":"2024-09-18T01:53:23.444116+0000","last_undegraded":"2024-09-18T01:53:23.444116+0000","last_fullsized":"2024-09-18T01:53:23.444116+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:26:58.168836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069378899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1147,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.443198+0000","last_change":"2024-09-18T01:16:55.498135+0000","last_active":"2024-09-18T01:53:23.443198+0000","last_peered":"2024-09-18T01:53:23.443198+0000","last_clean":"2024-09-18T01:53:23.443198+0000","last_became_active":"2024-09-18T01:16:55.497021+0000","last_became_peered":"2024-09-18T01:16:55.497021+0000","last_unstale":"2024-09-18T01:53:23.443198+0000","last_undegraded":"2024-09-18T01:53:23.443198+0000","last_fullsized":"2024-09-18T01:53:23.443198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_clean_scrub_stamp":"2024-09-18T01:16:54.479487+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:24:07.471201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1287352,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":1281470,"kb_avail":93084488,"statfs":{"total":96636764160,"available":95318515712,"internally_reserved":0,"allocated":5996544,"data_stored":2989833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1312225733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706105,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1544892,"kb_used_data":5928,"kb_used_omap":1,"kb_used_meta":1538942,"kb_avail":92826948,"statfs":{"total":96636764160,"available":95054794752,"internally_reserved":0,"allocated":6070272,"data_stored":3587324,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1575877061},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1423244,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":1417278,"kb_avail":92948596,"statfs":{"total":96636764160,"available":95179362304,"internally_reserved":0,"allocated":6086656,"data_stored":3593992,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1451293125},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:25.110 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-18T01:53:25.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T01:53:25.385 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:53:25.385 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-18T01:53:25.397 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-09-18T01:53:23.975925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4255488,"kb_used_data":17728,"kb_used_omap":4,"kb_used_meta":4237691,"kb_avail":278860032,"statfs":{"total":289910292480,"available":285552672768,"internally_reserved":0,"allocated":18153472,"data_stored":10171149,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4339395919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536832238,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-138025,"num_read_kb":-39828039,"num_write":-599093,"num_write_kb":-219651984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":-8,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002115"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.448253+0000","last_change":"2024-09-18T01:17:02.518768+0000","last_active":"2024-09-18T01:53:23.448253+0000","last_peered":"2024-09-18T01:53:23.448253+0000","last_clean":"2024-09-18T01:53:23.448253+0000","last_became_active":"2024-09-18T01:17:00.511441+0000","last_became_peered":"2024-09-18T01:17:00.511441+0000","last_unstale":"2024-09-18T01:53:23.448253+0000","last_undegraded":"2024-09-18T01:53:23.448253+0000","last_fullsized":"2024-09-18T01:53:23.448253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:23:40.338890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000687489,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.455003+0000","last_change":"2024-09-18T01:17:02.518477+0000","last_active":"2024-09-18T01:53:23.455003+0000","last_peered":"2024-09-18T01:53:23.455003+0000","last_clean":"2024-09-18T01:53:23.455003+0000","last_became_active":"2024-09-18T01:17:00.510105+0000","last_became_peered":"2024-09-18T01:17:00.510105+0000","last_unstale":"2024-09-18T01:53:23.455003+0000","last_undegraded":"2024-09-18T01:53:23.455003+0000","last_fullsized":"2024-09-18T01:53:23.455003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:12:46.477909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043520000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.445597+0000","last_change":"2024-09-18T01:17:02.519127+0000","last_active":"2024-09-18T01:53:23.445597+0000","last_peered":"2024-09-18T01:53:23.445597+0000","last_clean":"2024-09-18T01:53:23.445597+0000","last_became_active":"2024-09-18T01:17:00.509844+0000","last_became_peered":"2024-09-18T01:17:00.509844+0000","last_unstale":"2024-09-18T01:53:23.445597+0000","last_undegraded":"2024-09-18T01:53:23.445597+0000","last_fullsized":"2024-09-18T01:53:23.445597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:45:37.121984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.450865+0000","last_change":"2024-09-18T01:17:02.518480+0000","last_active":"2024-09-18T01:53:23.450865+0000","last_peered":"2024-09-18T01:53:23.450865+0000","last_clean":"2024-09-18T01:53:23.450865+0000","last_became_active":"2024-09-18T01:17:00.509856+0000","last_became_peered":"2024-09-18T01:17:00.509856+0000","last_unstale":"2024-09-18T01:53:23.450865+0000","last_undegraded":"2024-09-18T01:53:23.450865+0000","last_fullsized":"2024-09-18T01:53:23.450865+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:34:28.368843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042899300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1020,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.447312+0000","last_change":"2024-09-18T01:17:02.513682+0000","last_active":"2024-09-18T01:53:23.447312+0000","last_peered":"2024-09-18T01:53:23.447312+0000","last_clean":"2024-09-18T01:53:23.447312+0000","last_became_active":"2024-09-18T01:17:00.511011+0000","last_became_peered":"2024-09-18T01:17:00.511011+0000","last_unstale":"2024-09-18T01:53:23.447312+0000","last_undegraded":"2024-09-18T01:53:23.447312+0000","last_fullsized":"2024-09-18T01:53:23.447312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:14:05.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011029220000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1011,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-18T01:53:15.253798+0000","last_change":"2024-09-18T01:17:02.511032+0000","last_active":"2024-09-18T01:53:15.253798+0000","last_peered":"2024-09-18T01:53:15.253798+0000","last_clean":"2024-09-18T01:53:15.253798+0000","last_became_active":"2024-09-18T01:17:00.510439+0000","last_became_peered":"2024-09-18T01:17:00.510439+0000","last_unstale":"2024-09-18T01:53:15.253798+0000","last_undegraded":"2024-09-18T01:53:15.253798+0000","last_fullsized":"2024-09-18T01:53:15.253798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:04:11.663327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1012,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-18T01:53:15.260633+0000","last_change":"2024-09-18T01:17:02.510963+0000","last_active":"2024-09-18T01:53:15.260633+0000","last_peered":"2024-09-18T01:53:15.260633+0000","last_clean":"2024-09-18T01:53:15.260633+0000","last_became_active":"2024-09-18T01:17:00.512356+0000","last_became_peered":"2024-09-18T01:17:00.512356+0000","last_unstale":"2024-09-18T01:53:15.260633+0000","last_undegraded":"2024-09-18T01:53:15.260633+0000","last_fullsized":"2024-09-18T01:53:15.260633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:42:49.393044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1019,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.444116+0000","last_change":"2024-09-18T01:17:02.518778+0000","last_active":"2024-09-18T01:53:23.444116+0000","last_peered":"2024-09-18T01:53:23.444116+0000","last_clean":"2024-09-18T01:53:23.444116+0000","last_became_active":"2024-09-18T01:17:00.509327+0000","last_became_peered":"2024-09-18T01:17:00.509327+0000","last_unstale":"2024-09-18T01:53:23.444116+0000","last_undegraded":"2024-09-18T01:53:23.444116+0000","last_fullsized":"2024-09-18T01:53:23.444116+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:26:58.168836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069378899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1147,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.443198+0000","last_change":"2024-09-18T01:16:55.498135+0000","last_active":"2024-09-18T01:53:23.443198+0000","last_peered":"2024-09-18T01:53:23.443198+0000","last_clean":"2024-09-18T01:53:23.443198+0000","last_became_active":"2024-09-18T01:16:55.497021+0000","last_became_peered":"2024-09-18T01:16:55.497021+0000","last_unstale":"2024-09-18T01:53:23.443198+0000","last_undegraded":"2024-09-18T01:53:23.443198+0000","last_fullsized":"2024-09-18T01:53:23.443198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_clean_scrub_stamp":"2024-09-18T01:16:54.479487+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:24:07.471201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1287352,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":1281470,"kb_avail":93084488,"statfs":{"total":96636764160,"available":95318515712,"internally_reserved":0,"allocated":5996544,"data_stored":2989833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1312225733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706105,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1544892,"kb_used_data":5928,"kb_used_omap":1,"kb_used_meta":1538942,"kb_avail":92826948,"statfs":{"total":96636764160,"available":95054794752,"internally_reserved":0,"allocated":6070272,"data_stored":3587324,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1575877061},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1423244,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":1417278,"kb_avail":92948596,"statfs":{"total":96636764160,"available":95179362304,"internally_reserved":0,"allocated":6086656,"data_stored":3593992,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1451293125},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:25.398 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-18T01:53:25.672 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:53:25.673 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":507,"fsid":"fcf9d702-6ece-4878-bfd9-543b84db5aa5","created":"2024-09-18T01:16:43.930763+0000","modified":"2024-09-18T01:53:23.264546+0000","last_up_change":"2024-09-18T01:16:53.471910+0000","last_in_change":"2024-09-18T01:16:44.760042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T01:16:53.632560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T01:16:59.021351+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1629294-c19f-4008-ad1d-71ca6c8aa76e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6809","nonce":2571597248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6811","nonce":2571597248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6815","nonce":2571597248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2571597248},{"type":"v1","addr":"172.21.15.115:6813","nonce":2571597248}]},"public_addr":"172.21.15.115:6809/2571597248","cluster_addr":"172.21.15.115:6811/2571597248","heartbeat_back_addr":"172.21.15.115:6815/2571597248","heartbeat_front_addr":"172.21.15.115:6813/2571597248","state":["exists","up"]},{"osd":1,"uuid":"0b9f55f6-b049-401f-9d1a-9a7ff1167146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6817","nonce":1967664521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6819","nonce":1967664521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6823","nonce":1967664521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1967664521},{"type":"v1","addr":"172.21.15.115:6821","nonce":1967664521}]},"public_addr":"172.21.15.115:6817/1967664521","cluster_addr":"172.21.15.115:6819/1967664521","heartbeat_back_addr":"172.21.15.115:6823/1967664521","heartbeat_front_addr":"172.21.15.115:6821/1967664521","state":["exists","up"]},{"osd":2,"uuid":"0b29db8a-24e4-49bc-aac8-dbf6b6f2fa69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6801","nonce":1836880245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6803","nonce":1836880245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6807","nonce":1836880245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":1836880245},{"type":"v1","addr":"172.21.15.115:6805","nonce":1836880245}]},"public_addr":"172.21.15.115:6801/1836880245","cluster_addr":"172.21.15.115:6803/1836880245","heartbeat_back_addr":"172.21.15.115:6807/1836880245","heartbeat_front_addr":"172.21.15.115:6805/1836880245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:51.445385+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:51.218875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:50.713702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:0/2691271506":"2024-09-18T02:53:13.814561+0000","172.21.15.115:0/3140578018":"2024-09-18T02:51:45.437883+0000","172.21.15.115:0/3581269075":"2024-09-18T02:50:24.128584+0000","172.21.15.115:0/4261039767":"2024-09-18T02:48:55.077337+0000","172.21.15.115:0/175486709":"2024-09-18T02:39:12.000490+0000","172.21.15.115:0/3860791146":"2024-09-18T02:39:04.018871+0000","172.21.15.115:0/4281152448":"2024-09-18T02:25:33.894953+0000","172.21.15.115:0/771861480":"2024-09-18T02:46:58.799292+0000","172.21.15.115:0/1157743761":"2024-09-18T02:38:55.959328+0000","172.21.15.115:0/2291468494":"2024-09-18T02:40:46.871492+0000","172.21.15.115:0/2597954697":"2024-09-18T02:38:23.829278+0000","172.21.15.115:0/2624501050":"2024-09-18T02:37:59.772271+0000","172.21.15.115:0/3444014677":"2024-09-18T02:37:51.699710+0000","172.21.15.115:0/892801325":"2024-09-18T02:37:43.690696+0000","172.21.15.115:0/3768934402":"2024-09-18T02:37:27.057029+0000","172.21.15.115:0/45584950":"2024-09-18T02:41:27.088603+0000","172.21.15.115:0/4036845891":"2024-09-18T02:43:06.883729+0000","172.21.15.115:0/1247349145":"2024-09-18T02:38:15.791189+0000","172.21.15.115:0/1726224199":"2024-09-18T02:37:11.000781+0000","172.21.15.115:0/2056836318":"2024-09-18T02:26:22.466840+0000","172.21.15.115:0/3312509792":"2024-09-18T02:41:51.334272+0000","172.21.15.115:0/2703083048":"2024-09-18T02:21:09.883307+0000","172.21.15.115:0/1959955918":"2024-09-18T02:42:50.764961+0000","172.21.15.115:0/3215071298":"2024-09-18T02:49:40.128673+0000","172.21.15.115:0/2766344395":"2024-09-18T02:22:02.344722+0000","172.21.15.115:0/2959185685":"2024-09-18T02:27:17.610991+0000","172.21.15.115:0/1295826306":"2024-09-18T02:22:58.832694+0000","172.21.15.115:0/4196959691":"2024-09-18T02:37:35.712334+0000","172.21.15.115:0/1059303099":"2024-09-18T02:38:39.846240+0000","172.21.15.115:0/4033169519":"2024-09-18T02:23:50.615992+0000","172.21.15.115:0/258977725":"2024-09-18T02:42:09.050659+0000","172.21.15.115:0/355176106":"2024-09-18T02:40:54.987703+0000","172.21.15.115:0/1430190208":"2024-09-18T02:24:44.987843+0000","172.21.15.115:0/3502332694":"2024-09-18T02:36:54.178075+0000","172.21.15.115:0/1253679982":"2024-09-18T02:38:47.897732+0000","172.21.15.115:0/380449980":"2024-09-18T02:39:42.080917+0000","172.21.15.115:0/379589041":"2024-09-18T02:42:17.233472+0000","172.21.15.115:0/2491925255":"2024-09-18T02:52:27.174910+0000","172.21.15.115:0/2153763265":"2024-09-18T02:36:45.304357+0000","172.21.15.115:0/1487881211":"2024-09-18T02:39:50.754150+0000","172.21.15.115:0/1811374424":"2024-09-18T02:48:17.314125+0000","172.21.15.115:0/1932175127":"2024-09-18T02:19:30.776938+0000","172.21.15.115:0/449389481":"2024-09-18T02:37:03.002640+0000","172.21.15.115:0/3779840221":"2024-09-18T02:37:18.970744+0000","172.21.15.115:0/2177065516":"2024-09-18T02:39:20.101427+0000","172.21.15.115:0/3619184685":"2024-09-18T02:39:35.037262+0000","172.21.15.115:0/872944865":"2024-09-18T02:43:30.806842+0000","172.21.15.115:0/353576370":"2024-09-18T02:39:58.757208+0000","172.21.15.115:0/3858266263":"2024-09-18T02:41:35.100848+0000","172.21.15.115:0/4159473717":"2024-09-18T02:42:34.149482+0000","172.21.15.115:0/2420005007":"2024-09-18T02:43:14.818972+0000","172.21.15.115:0/1396710972":"2024-09-18T02:40:06.784006+0000","172.21.15.115:0/2588378561":"2024-09-18T02:41:11.021537+0000","172.21.15.115:0/1564549034":"2024-09-18T02:38:31.840250+0000","172.21.15.115:0/3750075060":"2024-09-18T02:42:42.123618+0000","172.21.15.115:0/78035378":"2024-09-18T02:50:59.706730+0000","172.21.15.115:0/400160315":"2024-09-18T02:47:38.909959+0000","172.21.15.115:0/1341056242":"2024-09-18T02:40:14.770595+0000","172.21.15.115:0/2203798337":"2024-09-18T02:40:22.847837+0000","172.21.15.115:0/3309660672":"2024-09-18T02:40:30.856651+0000","172.21.15.115:0/2335056525":"2024-09-18T02:39:28.076281+0000","172.21.15.115:0/2514030157":"2024-09-18T02:41:19.017996+0000","172.21.15.115:0/592611192":"2024-09-18T02:40:38.966411+0000","172.21.15.115:0/3854743356":"2024-09-18T02:41:03.016362+0000","172.21.15.115:0/1575262947":"2024-09-18T02:20:19.438296+0000","172.21.15.115:0/3852713521":"2024-09-18T02:41:43.107014+0000","172.21.15.115:0/2776337937":"2024-09-18T02:42:00.212364+0000","172.21.15.115:0/1566733925":"2024-09-18T02:38:07.760778+0000","172.21.15.115:0/1647153475":"2024-09-18T02:42:26.074024+0000","172.21.15.115:0/3625302055":"2024-09-18T02:43:22.809631+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":194,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:53:26.686 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-18T01:53:26.686 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T01:53:26.819 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-18T01:53:26.819 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T01:53:26.819 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-18T01:53:26.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-18T01:53:27.106 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T01:53:27.119 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-18T01:53:27.119 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T01:53:27.240 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-18T01:53:27.241 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T01:53:27.241 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-18T01:53:27.252 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-18T01:53:27.527 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T01:53:27.540 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-18T01:53:27.540 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T01:53:27.662 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-18T01:53:27.662 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T01:53:27.662 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-18T01:53:27.673 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-18T01:53:27.947 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T01:53:27.960 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T01:53:28.231 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:53:28.231 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-18T01:53:28.243 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":1602,"stamp":"2024-09-18T01:53:27.976825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":150,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4255512,"kb_used_data":17752,"kb_used_omap":4,"kb_used_meta":4237691,"kb_avail":278860008,"statfs":{"total":289910292480,"available":285552648192,"internally_reserved":0,"allocated":18178048,"data_stored":10184921,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4339395919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-138258,"num_read_kb":-39828243,"num_write":-599266,"num_write_kb":-219651995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":-8,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002180"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.448253+0000","last_change":"2024-09-18T01:17:02.518768+0000","last_active":"2024-09-18T01:53:23.448253+0000","last_peered":"2024-09-18T01:53:23.448253+0000","last_clean":"2024-09-18T01:53:23.448253+0000","last_became_active":"2024-09-18T01:17:00.511441+0000","last_became_peered":"2024-09-18T01:17:00.511441+0000","last_unstale":"2024-09-18T01:53:23.448253+0000","last_undegraded":"2024-09-18T01:53:23.448253+0000","last_fullsized":"2024-09-18T01:53:23.448253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:23:40.338890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000687489,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.455003+0000","last_change":"2024-09-18T01:17:02.518477+0000","last_active":"2024-09-18T01:53:23.455003+0000","last_peered":"2024-09-18T01:53:23.455003+0000","last_clean":"2024-09-18T01:53:23.455003+0000","last_became_active":"2024-09-18T01:17:00.510105+0000","last_became_peered":"2024-09-18T01:17:00.510105+0000","last_unstale":"2024-09-18T01:53:23.455003+0000","last_undegraded":"2024-09-18T01:53:23.455003+0000","last_fullsized":"2024-09-18T01:53:23.455003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:12:46.477909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043520000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.445597+0000","last_change":"2024-09-18T01:17:02.519127+0000","last_active":"2024-09-18T01:53:23.445597+0000","last_peered":"2024-09-18T01:53:23.445597+0000","last_clean":"2024-09-18T01:53:23.445597+0000","last_became_active":"2024-09-18T01:17:00.509844+0000","last_became_peered":"2024-09-18T01:17:00.509844+0000","last_unstale":"2024-09-18T01:53:23.445597+0000","last_undegraded":"2024-09-18T01:53:23.445597+0000","last_fullsized":"2024-09-18T01:53:23.445597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:45:37.121984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.450865+0000","last_change":"2024-09-18T01:17:02.518480+0000","last_active":"2024-09-18T01:53:23.450865+0000","last_peered":"2024-09-18T01:53:23.450865+0000","last_clean":"2024-09-18T01:53:23.450865+0000","last_became_active":"2024-09-18T01:17:00.509856+0000","last_became_peered":"2024-09-18T01:17:00.509856+0000","last_unstale":"2024-09-18T01:53:23.450865+0000","last_undegraded":"2024-09-18T01:53:23.450865+0000","last_fullsized":"2024-09-18T01:53:23.450865+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:34:28.368843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042899300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1020,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.447312+0000","last_change":"2024-09-18T01:17:02.513682+0000","last_active":"2024-09-18T01:53:23.447312+0000","last_peered":"2024-09-18T01:53:23.447312+0000","last_clean":"2024-09-18T01:53:23.447312+0000","last_became_active":"2024-09-18T01:17:00.511011+0000","last_became_peered":"2024-09-18T01:17:00.511011+0000","last_unstale":"2024-09-18T01:53:23.447312+0000","last_undegraded":"2024-09-18T01:53:23.447312+0000","last_fullsized":"2024-09-18T01:53:23.447312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:14:05.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011029220000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1015,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.272003+0000","last_change":"2024-09-18T01:17:02.511032+0000","last_active":"2024-09-18T01:53:23.272003+0000","last_peered":"2024-09-18T01:53:23.272003+0000","last_clean":"2024-09-18T01:53:23.272003+0000","last_became_active":"2024-09-18T01:17:00.510439+0000","last_became_peered":"2024-09-18T01:17:00.510439+0000","last_unstale":"2024-09-18T01:53:23.272003+0000","last_undegraded":"2024-09-18T01:53:23.272003+0000","last_fullsized":"2024-09-18T01:53:23.272003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:04:11.663327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.271133+0000","last_change":"2024-09-18T01:17:02.510963+0000","last_active":"2024-09-18T01:53:23.271133+0000","last_peered":"2024-09-18T01:53:23.271133+0000","last_clean":"2024-09-18T01:53:23.271133+0000","last_became_active":"2024-09-18T01:17:00.512356+0000","last_became_peered":"2024-09-18T01:17:00.512356+0000","last_unstale":"2024-09-18T01:53:23.271133+0000","last_undegraded":"2024-09-18T01:53:23.271133+0000","last_fullsized":"2024-09-18T01:53:23.271133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:42:49.393044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1019,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.444116+0000","last_change":"2024-09-18T01:17:02.518778+0000","last_active":"2024-09-18T01:53:23.444116+0000","last_peered":"2024-09-18T01:53:23.444116+0000","last_clean":"2024-09-18T01:53:23.444116+0000","last_became_active":"2024-09-18T01:17:00.509327+0000","last_became_peered":"2024-09-18T01:17:00.509327+0000","last_unstale":"2024-09-18T01:53:23.444116+0000","last_undegraded":"2024-09-18T01:53:23.444116+0000","last_fullsized":"2024-09-18T01:53:23.444116+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:59.489577+0000","last_clean_scrub_stamp":"2024-09-18T01:16:59.489577+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:26:58.168836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069378899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1147,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:23.443198+0000","last_change":"2024-09-18T01:16:55.498135+0000","last_active":"2024-09-18T01:53:23.443198+0000","last_peered":"2024-09-18T01:53:23.443198+0000","last_clean":"2024-09-18T01:53:23.443198+0000","last_became_active":"2024-09-18T01:16:55.497021+0000","last_became_peered":"2024-09-18T01:16:55.497021+0000","last_unstale":"2024-09-18T01:53:23.443198+0000","last_undegraded":"2024-09-18T01:53:23.443198+0000","last_fullsized":"2024-09-18T01:53:23.443198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:54.479487+0000","last_clean_scrub_stamp":"2024-09-18T01:16:54.479487+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:24:07.471201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1287376,"kb_used_data":5880,"kb_used_omap":1,"kb_used_meta":1281470,"kb_avail":93084464,"statfs":{"total":96636764160,"available":95318491136,"internally_reserved":0,"allocated":6021120,"data_stored":3003605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1312225733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706105,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1544892,"kb_used_data":5928,"kb_used_omap":1,"kb_used_meta":1538942,"kb_avail":92826948,"statfs":{"total":96636764160,"available":95054794752,"internally_reserved":0,"allocated":6070272,"data_stored":3587324,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1575877061},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1423244,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":1417278,"kb_avail":92948596,"statfs":{"total":96636764160,"available":95179362304,"internally_reserved":0,"allocated":6086656,"data_stored":3593992,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1451293125},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:28.246 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-18T01:16:59.489577+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.247 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-18T01:16:59.489577+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.247 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-18T01:16:59.489577+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.247 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T01:16:59.489577+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.247 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-18T01:16:59.489577+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.247 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-18T01:16:59.489577+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.247 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-18T01:16:59.489577+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.247 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-18T01:16:59.489577+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.247 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T01:16:54.479487+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=54, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=25, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:28.248 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T01:53:48.248 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T01:53:48.533 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T01:53:48.533 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-18T01:53:48.545 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":1612,"stamp":"2024-09-18T01:53:47.980493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4255512,"kb_used_data":17768,"kb_used_omap":4,"kb_used_meta":4237691,"kb_avail":278860008,"statfs":{"total":289910292480,"available":285552648192,"internally_reserved":0,"allocated":18194432,"data_stored":10191589,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4339395919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002087"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1026,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:32.510714+0000","last_change":"2024-09-18T01:53:32.510740+0000","last_active":"2024-09-18T01:53:32.510714+0000","last_peered":"2024-09-18T01:53:32.510714+0000","last_clean":"2024-09-18T01:53:32.510714+0000","last_became_active":"2024-09-18T01:17:00.511441+0000","last_became_peered":"2024-09-18T01:17:00.511441+0000","last_unstale":"2024-09-18T01:53:32.510714+0000","last_undegraded":"2024-09-18T01:53:32.510714+0000","last_fullsized":"2024-09-18T01:53:32.510714+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:32.510612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:32.510612+0000","last_clean_scrub_stamp":"2024-09-18T01:53:32.510612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:02:13.862760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000687489,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1026,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:29.554160+0000","last_change":"2024-09-18T01:53:29.554183+0000","last_active":"2024-09-18T01:53:29.554160+0000","last_peered":"2024-09-18T01:53:29.554160+0000","last_clean":"2024-09-18T01:53:29.554160+0000","last_became_active":"2024-09-18T01:17:00.510105+0000","last_became_peered":"2024-09-18T01:17:00.510105+0000","last_unstale":"2024-09-18T01:53:29.554160+0000","last_undegraded":"2024-09-18T01:53:29.554160+0000","last_fullsized":"2024-09-18T01:53:29.554160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:29.554034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:29.554034+0000","last_clean_scrub_stamp":"2024-09-18T01:53:29.554034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:11:50.573229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043520000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1027,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:31.495010+0000","last_change":"2024-09-18T01:53:31.495032+0000","last_active":"2024-09-18T01:53:31.495010+0000","last_peered":"2024-09-18T01:53:31.495010+0000","last_clean":"2024-09-18T01:53:31.495010+0000","last_became_active":"2024-09-18T01:17:00.509844+0000","last_became_peered":"2024-09-18T01:17:00.509844+0000","last_unstale":"2024-09-18T01:53:31.495010+0000","last_undegraded":"2024-09-18T01:53:31.495010+0000","last_fullsized":"2024-09-18T01:53:31.495010+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:31.494916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:31.494916+0000","last_clean_scrub_stamp":"2024-09-18T01:53:31.494916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:20:04.816934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1027,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:30.508765+0000","last_change":"2024-09-18T01:53:30.508790+0000","last_active":"2024-09-18T01:53:30.508765+0000","last_peered":"2024-09-18T01:53:30.508765+0000","last_clean":"2024-09-18T01:53:30.508765+0000","last_became_active":"2024-09-18T01:17:00.509856+0000","last_became_peered":"2024-09-18T01:17:00.509856+0000","last_unstale":"2024-09-18T01:53:30.508765+0000","last_undegraded":"2024-09-18T01:53:30.508765+0000","last_fullsized":"2024-09-18T01:53:30.508765+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:30.508672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:30.508672+0000","last_clean_scrub_stamp":"2024-09-18T01:53:30.508672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:45:38.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042899300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"507'3","reported_seq":1030,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:27.248694+0000","last_change":"2024-09-18T01:53:27.248718+0000","last_active":"2024-09-18T01:53:27.248694+0000","last_peered":"2024-09-18T01:53:27.248694+0000","last_clean":"2024-09-18T01:53:27.248694+0000","last_became_active":"2024-09-18T01:17:00.511011+0000","last_became_peered":"2024-09-18T01:17:00.511011+0000","last_unstale":"2024-09-18T01:53:27.248694+0000","last_undegraded":"2024-09-18T01:53:27.248694+0000","last_fullsized":"2024-09-18T01:53:27.248694+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'3","last_scrub_stamp":"2024-09-18T01:53:27.248622+0000","last_deep_scrub":"507'3","last_deep_scrub_stamp":"2024-09-18T01:53:27.248622+0000","last_clean_scrub_stamp":"2024-09-18T01:53:27.248622+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:27:19.021224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011029220000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1024,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:28.425689+0000","last_change":"2024-09-18T01:53:28.425716+0000","last_active":"2024-09-18T01:53:28.425689+0000","last_peered":"2024-09-18T01:53:28.425689+0000","last_clean":"2024-09-18T01:53:28.425689+0000","last_became_active":"2024-09-18T01:17:00.510439+0000","last_became_peered":"2024-09-18T01:17:00.510439+0000","last_unstale":"2024-09-18T01:53:28.425689+0000","last_undegraded":"2024-09-18T01:53:28.425689+0000","last_fullsized":"2024-09-18T01:53:28.425689+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:28.425605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:28.425605+0000","last_clean_scrub_stamp":"2024-09-18T01:53:28.425605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:12:56.222124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1025,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:29.433138+0000","last_change":"2024-09-18T01:53:29.433187+0000","last_active":"2024-09-18T01:53:29.433138+0000","last_peered":"2024-09-18T01:53:29.433138+0000","last_clean":"2024-09-18T01:53:29.433138+0000","last_became_active":"2024-09-18T01:17:00.512356+0000","last_became_peered":"2024-09-18T01:17:00.512356+0000","last_unstale":"2024-09-18T01:53:29.433138+0000","last_undegraded":"2024-09-18T01:53:29.433138+0000","last_fullsized":"2024-09-18T01:53:29.433138+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:29.433022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:29.433022+0000","last_clean_scrub_stamp":"2024-09-18T01:53:29.433022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:56:49.695617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000426014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"507'2","reported_seq":1029,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:28.593699+0000","last_change":"2024-09-18T01:53:28.593727+0000","last_active":"2024-09-18T01:53:28.593699+0000","last_peered":"2024-09-18T01:53:28.593699+0000","last_clean":"2024-09-18T01:53:28.593699+0000","last_became_active":"2024-09-18T01:17:00.509327+0000","last_became_peered":"2024-09-18T01:17:00.509327+0000","last_unstale":"2024-09-18T01:53:28.593699+0000","last_undegraded":"2024-09-18T01:53:28.593699+0000","last_fullsized":"2024-09-18T01:53:28.593699+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-09-18T01:53:28.593611+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-09-18T01:53:28.593611+0000","last_clean_scrub_stamp":"2024-09-18T01:53:28.593611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:37:51.941955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069378899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"507'99","reported_seq":1158,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-18T01:53:27.598485+0000","last_change":"2024-09-18T01:53:27.598515+0000","last_active":"2024-09-18T01:53:27.598485+0000","last_peered":"2024-09-18T01:53:27.598485+0000","last_clean":"2024-09-18T01:53:27.598485+0000","last_became_active":"2024-09-18T01:16:55.497021+0000","last_became_peered":"2024-09-18T01:16:55.497021+0000","last_unstale":"2024-09-18T01:53:27.598485+0000","last_undegraded":"2024-09-18T01:53:27.598485+0000","last_fullsized":"2024-09-18T01:53:27.598485+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'99","last_scrub_stamp":"2024-09-18T01:53:27.598414+0000","last_deep_scrub":"507'99","last_deep_scrub_stamp":"2024-09-18T01:53:27.598414+0000","last_clean_scrub_stamp":"2024-09-18T01:53:27.598414+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:44:01.905881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59938,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1287368,"kb_used_data":5880,"kb_used_omap":1,"kb_used_meta":1281470,"kb_avail":93084472,"statfs":{"total":96636764160,"available":95318499328,"internally_reserved":0,"allocated":6021120,"data_stored":3003605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1312225733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706109,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1544904,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":1538942,"kb_avail":92826936,"statfs":{"total":96636764160,"available":95054782464,"internally_reserved":0,"allocated":6086656,"data_stored":3593992,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1575877061},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1423240,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":1417278,"kb_avail":92948600,"statfs":{"total":96636764160,"available":95179366400,"internally_reserved":0,"allocated":6086656,"data_stored":3593992,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1451293125},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29969,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:48.547 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-18T01:53:48.838 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T01:53:48.838 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T01:53:48.838 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-18T01:53:48.838 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:48.899 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-18T01:53:48.900 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-18T01:53:48.900 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:48.988 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-18T01:53:48.988 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-18T01:53:48.988 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:49.066 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-18T01:53:49.066 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T01:53:49.066 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-18T01:53:49.066 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:49.096 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-18T01:53:49.096 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T01:53:49.096 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-18T01:53:49.096 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:53:49.109 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-18T01:53:49.109 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T01:53:49.109 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-18T01:53:49.207 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:53:49.207 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-18T01:53:49.467 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:53:49.467 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-18T01:53:49.593 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:53:49.593 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-18T01:53:49.729 INFO:tasks.ceph:Archiving mon data... 2024-09-18T01:53:49.731 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706/data/cluster2.mon.a.tgz 2024-09-18T01:53:49.731 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-09-18T01:53:49.746 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.PcUDYyMYlE 2024-09-18T01:53:49.746 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.PcUDYyMYlE 2024-09-18T01:53:49.905 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.PcUDYyMYlE 2024-09-18T01:53:50.006 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.PcUDYyMYlE is 412KB 2024-09-18T01:53:50.082 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.PcUDYyMYlE 2024-09-18T01:53:50.098 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T01:53:50.098 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T01:53:50.101 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T01:53:50.176 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T01:53:50.188 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-18T01:53:50.188 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T01:53:50.488 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:53:50.572 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-09-18T01:53:50.572 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2024-09-18T01:53:48.566391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":450580,"kb_used_data":11480,"kb_used_omap":4,"kb_used_meta":439035,"kb_avail":282664940,"statfs":{"total":289910292480,"available":289448898560,"internally_reserved":0,"allocated":11755520,"data_stored":7394692,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":449572175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002199"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.630324+0000","last_change":"2024-09-18T01:16:24.048908+0000","last_active":"2024-09-18T01:53:25.630324+0000","last_peered":"2024-09-18T01:53:25.630324+0000","last_clean":"2024-09-18T01:53:25.630324+0000","last_became_active":"2024-09-18T01:16:22.049384+0000","last_became_peered":"2024-09-18T01:16:22.049384+0000","last_unstale":"2024-09-18T01:53:25.630324+0000","last_undegraded":"2024-09-18T01:53:25.630324+0000","last_fullsized":"2024-09-18T01:53:25.630324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:08:44.387269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067731499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.634629+0000","last_change":"2024-09-18T01:16:24.049185+0000","last_active":"2024-09-18T01:53:25.634629+0000","last_peered":"2024-09-18T01:53:25.634629+0000","last_clean":"2024-09-18T01:53:25.634629+0000","last_became_active":"2024-09-18T01:16:22.048383+0000","last_became_peered":"2024-09-18T01:16:22.048383+0000","last_unstale":"2024-09-18T01:53:25.634629+0000","last_undegraded":"2024-09-18T01:53:25.634629+0000","last_fullsized":"2024-09-18T01:53:25.634629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:33:41.916036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076761799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.633082+0000","last_change":"2024-09-18T01:16:24.048791+0000","last_active":"2024-09-18T01:53:25.633082+0000","last_peered":"2024-09-18T01:53:25.633082+0000","last_clean":"2024-09-18T01:53:25.633082+0000","last_became_active":"2024-09-18T01:16:22.049251+0000","last_became_peered":"2024-09-18T01:16:22.049251+0000","last_unstale":"2024-09-18T01:53:25.633082+0000","last_undegraded":"2024-09-18T01:53:25.633082+0000","last_fullsized":"2024-09-18T01:53:25.633082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:25.091421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057751099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.630903+0000","last_change":"2024-09-18T01:16:24.049406+0000","last_active":"2024-09-18T01:53:25.630903+0000","last_peered":"2024-09-18T01:53:25.630903+0000","last_clean":"2024-09-18T01:53:25.630903+0000","last_became_active":"2024-09-18T01:16:22.049287+0000","last_became_peered":"2024-09-18T01:16:22.049287+0000","last_unstale":"2024-09-18T01:53:25.630903+0000","last_undegraded":"2024-09-18T01:53:25.630903+0000","last_fullsized":"2024-09-18T01:53:25.630903+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:51:35.915729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001186821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:21.988579+0000","last_change":"2024-09-18T01:16:24.045936+0000","last_active":"2024-09-18T01:53:21.988579+0000","last_peered":"2024-09-18T01:53:21.988579+0000","last_clean":"2024-09-18T01:53:21.988579+0000","last_became_active":"2024-09-18T01:16:22.048463+0000","last_became_peered":"2024-09-18T01:16:22.048463+0000","last_unstale":"2024-09-18T01:53:21.988579+0000","last_undegraded":"2024-09-18T01:53:21.988579+0000","last_fullsized":"2024-09-18T01:53:21.988579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:09:27.308691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050943500000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.881863+0000","last_change":"2024-09-18T01:16:24.060226+0000","last_active":"2024-09-18T01:53:25.881863+0000","last_peered":"2024-09-18T01:53:25.881863+0000","last_clean":"2024-09-18T01:53:25.881863+0000","last_became_active":"2024-09-18T01:16:22.048849+0000","last_became_peered":"2024-09-18T01:16:22.048849+0000","last_unstale":"2024-09-18T01:53:25.881863+0000","last_undegraded":"2024-09-18T01:53:25.881863+0000","last_fullsized":"2024-09-18T01:53:25.881863+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:32.872881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056915500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.881668+0000","last_change":"2024-09-18T01:16:24.060127+0000","last_active":"2024-09-18T01:53:25.881668+0000","last_peered":"2024-09-18T01:53:25.881668+0000","last_clean":"2024-09-18T01:53:25.881668+0000","last_became_active":"2024-09-18T01:16:22.045793+0000","last_became_peered":"2024-09-18T01:16:22.045793+0000","last_unstale":"2024-09-18T01:53:25.881668+0000","last_undegraded":"2024-09-18T01:53:25.881668+0000","last_fullsized":"2024-09-18T01:53:25.881668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:08:15.672511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048395300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.633940+0000","last_change":"2024-09-18T01:16:24.049520+0000","last_active":"2024-09-18T01:53:25.633940+0000","last_peered":"2024-09-18T01:53:25.633940+0000","last_clean":"2024-09-18T01:53:25.633940+0000","last_became_active":"2024-09-18T01:16:22.045810+0000","last_became_peered":"2024-09-18T01:16:22.045810+0000","last_unstale":"2024-09-18T01:53:25.633940+0000","last_undegraded":"2024-09-18T01:53:25.633940+0000","last_fullsized":"2024-09-18T01:53:25.633940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:33.391698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010551429999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1075,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.628762+0000","last_change":"2024-09-18T01:16:17.153205+0000","last_active":"2024-09-18T01:53:25.628762+0000","last_peered":"2024-09-18T01:53:25.628762+0000","last_clean":"2024-09-18T01:53:25.628762+0000","last_became_active":"2024-09-18T01:16:17.151939+0000","last_became_peered":"2024-09-18T01:16:17.151939+0000","last_unstale":"2024-09-18T01:53:25.628762+0000","last_undegraded":"2024-09-18T01:53:25.628762+0000","last_fullsized":"2024-09-18T01:53:25.628762+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_clean_scrub_stamp":"2024-09-18T01:16:16.017299+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:51:08.601096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706117,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151580,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220260,"statfs":{"total":96636764160,"available":96481546240,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706118,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165532,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206308,"statfs":{"total":96636764160,"available":96467259392,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133468,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94238372,"statfs":{"total":96636764160,"available":96500092928,"internally_reserved":0,"allocated":3874816,"data_stored":2071306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132774341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:50.573 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T01:53:50.859 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:53:50.860 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-09-18T01:53:50.871 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2024-09-18T01:53:50.566805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":450580,"kb_used_data":11480,"kb_used_omap":4,"kb_used_meta":439035,"kb_avail":282664940,"statfs":{"total":289910292480,"available":289448898560,"internally_reserved":0,"allocated":11755520,"data_stored":7394692,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":449572175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002200"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.630324+0000","last_change":"2024-09-18T01:16:24.048908+0000","last_active":"2024-09-18T01:53:25.630324+0000","last_peered":"2024-09-18T01:53:25.630324+0000","last_clean":"2024-09-18T01:53:25.630324+0000","last_became_active":"2024-09-18T01:16:22.049384+0000","last_became_peered":"2024-09-18T01:16:22.049384+0000","last_unstale":"2024-09-18T01:53:25.630324+0000","last_undegraded":"2024-09-18T01:53:25.630324+0000","last_fullsized":"2024-09-18T01:53:25.630324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:08:44.387269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067731499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.634629+0000","last_change":"2024-09-18T01:16:24.049185+0000","last_active":"2024-09-18T01:53:25.634629+0000","last_peered":"2024-09-18T01:53:25.634629+0000","last_clean":"2024-09-18T01:53:25.634629+0000","last_became_active":"2024-09-18T01:16:22.048383+0000","last_became_peered":"2024-09-18T01:16:22.048383+0000","last_unstale":"2024-09-18T01:53:25.634629+0000","last_undegraded":"2024-09-18T01:53:25.634629+0000","last_fullsized":"2024-09-18T01:53:25.634629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:33:41.916036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076761799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.633082+0000","last_change":"2024-09-18T01:16:24.048791+0000","last_active":"2024-09-18T01:53:25.633082+0000","last_peered":"2024-09-18T01:53:25.633082+0000","last_clean":"2024-09-18T01:53:25.633082+0000","last_became_active":"2024-09-18T01:16:22.049251+0000","last_became_peered":"2024-09-18T01:16:22.049251+0000","last_unstale":"2024-09-18T01:53:25.633082+0000","last_undegraded":"2024-09-18T01:53:25.633082+0000","last_fullsized":"2024-09-18T01:53:25.633082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:25.091421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057751099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.630903+0000","last_change":"2024-09-18T01:16:24.049406+0000","last_active":"2024-09-18T01:53:25.630903+0000","last_peered":"2024-09-18T01:53:25.630903+0000","last_clean":"2024-09-18T01:53:25.630903+0000","last_became_active":"2024-09-18T01:16:22.049287+0000","last_became_peered":"2024-09-18T01:16:22.049287+0000","last_unstale":"2024-09-18T01:53:25.630903+0000","last_undegraded":"2024-09-18T01:53:25.630903+0000","last_fullsized":"2024-09-18T01:53:25.630903+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:51:35.915729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001186821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:21.988579+0000","last_change":"2024-09-18T01:16:24.045936+0000","last_active":"2024-09-18T01:53:21.988579+0000","last_peered":"2024-09-18T01:53:21.988579+0000","last_clean":"2024-09-18T01:53:21.988579+0000","last_became_active":"2024-09-18T01:16:22.048463+0000","last_became_peered":"2024-09-18T01:16:22.048463+0000","last_unstale":"2024-09-18T01:53:21.988579+0000","last_undegraded":"2024-09-18T01:53:21.988579+0000","last_fullsized":"2024-09-18T01:53:21.988579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:09:27.308691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050943500000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.881863+0000","last_change":"2024-09-18T01:16:24.060226+0000","last_active":"2024-09-18T01:53:25.881863+0000","last_peered":"2024-09-18T01:53:25.881863+0000","last_clean":"2024-09-18T01:53:25.881863+0000","last_became_active":"2024-09-18T01:16:22.048849+0000","last_became_peered":"2024-09-18T01:16:22.048849+0000","last_unstale":"2024-09-18T01:53:25.881863+0000","last_undegraded":"2024-09-18T01:53:25.881863+0000","last_fullsized":"2024-09-18T01:53:25.881863+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:32.872881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056915500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.881668+0000","last_change":"2024-09-18T01:16:24.060127+0000","last_active":"2024-09-18T01:53:25.881668+0000","last_peered":"2024-09-18T01:53:25.881668+0000","last_clean":"2024-09-18T01:53:25.881668+0000","last_became_active":"2024-09-18T01:16:22.045793+0000","last_became_peered":"2024-09-18T01:16:22.045793+0000","last_unstale":"2024-09-18T01:53:25.881668+0000","last_undegraded":"2024-09-18T01:53:25.881668+0000","last_fullsized":"2024-09-18T01:53:25.881668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:08:15.672511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048395300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.633940+0000","last_change":"2024-09-18T01:16:24.049520+0000","last_active":"2024-09-18T01:53:25.633940+0000","last_peered":"2024-09-18T01:53:25.633940+0000","last_clean":"2024-09-18T01:53:25.633940+0000","last_became_active":"2024-09-18T01:16:22.045810+0000","last_became_peered":"2024-09-18T01:16:22.045810+0000","last_unstale":"2024-09-18T01:53:25.633940+0000","last_undegraded":"2024-09-18T01:53:25.633940+0000","last_fullsized":"2024-09-18T01:53:25.633940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:33.391698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010551429999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1075,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.628762+0000","last_change":"2024-09-18T01:16:17.153205+0000","last_active":"2024-09-18T01:53:25.628762+0000","last_peered":"2024-09-18T01:53:25.628762+0000","last_clean":"2024-09-18T01:53:25.628762+0000","last_became_active":"2024-09-18T01:16:17.151939+0000","last_became_peered":"2024-09-18T01:16:17.151939+0000","last_unstale":"2024-09-18T01:53:25.628762+0000","last_undegraded":"2024-09-18T01:53:25.628762+0000","last_fullsized":"2024-09-18T01:53:25.628762+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_clean_scrub_stamp":"2024-09-18T01:16:16.017299+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:51:08.601096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706117,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151580,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220260,"statfs":{"total":96636764160,"available":96481546240,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706118,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165532,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206308,"statfs":{"total":96636764160,"available":96467259392,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133468,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94238372,"statfs":{"total":96636764160,"available":96500092928,"internally_reserved":0,"allocated":3874816,"data_stored":2071306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132774341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:50.872 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-18T01:53:50.872 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T01:53:51.157 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:53:51.157 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-09-18T01:53:51.168 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2024-09-18T01:53:50.566805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":450580,"kb_used_data":11480,"kb_used_omap":4,"kb_used_meta":439035,"kb_avail":282664940,"statfs":{"total":289910292480,"available":289448898560,"internally_reserved":0,"allocated":11755520,"data_stored":7394692,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":449572175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002200"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.630324+0000","last_change":"2024-09-18T01:16:24.048908+0000","last_active":"2024-09-18T01:53:25.630324+0000","last_peered":"2024-09-18T01:53:25.630324+0000","last_clean":"2024-09-18T01:53:25.630324+0000","last_became_active":"2024-09-18T01:16:22.049384+0000","last_became_peered":"2024-09-18T01:16:22.049384+0000","last_unstale":"2024-09-18T01:53:25.630324+0000","last_undegraded":"2024-09-18T01:53:25.630324+0000","last_fullsized":"2024-09-18T01:53:25.630324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:08:44.387269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067731499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.634629+0000","last_change":"2024-09-18T01:16:24.049185+0000","last_active":"2024-09-18T01:53:25.634629+0000","last_peered":"2024-09-18T01:53:25.634629+0000","last_clean":"2024-09-18T01:53:25.634629+0000","last_became_active":"2024-09-18T01:16:22.048383+0000","last_became_peered":"2024-09-18T01:16:22.048383+0000","last_unstale":"2024-09-18T01:53:25.634629+0000","last_undegraded":"2024-09-18T01:53:25.634629+0000","last_fullsized":"2024-09-18T01:53:25.634629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:33:41.916036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076761799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.633082+0000","last_change":"2024-09-18T01:16:24.048791+0000","last_active":"2024-09-18T01:53:25.633082+0000","last_peered":"2024-09-18T01:53:25.633082+0000","last_clean":"2024-09-18T01:53:25.633082+0000","last_became_active":"2024-09-18T01:16:22.049251+0000","last_became_peered":"2024-09-18T01:16:22.049251+0000","last_unstale":"2024-09-18T01:53:25.633082+0000","last_undegraded":"2024-09-18T01:53:25.633082+0000","last_fullsized":"2024-09-18T01:53:25.633082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:25.091421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057751099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.630903+0000","last_change":"2024-09-18T01:16:24.049406+0000","last_active":"2024-09-18T01:53:25.630903+0000","last_peered":"2024-09-18T01:53:25.630903+0000","last_clean":"2024-09-18T01:53:25.630903+0000","last_became_active":"2024-09-18T01:16:22.049287+0000","last_became_peered":"2024-09-18T01:16:22.049287+0000","last_unstale":"2024-09-18T01:53:25.630903+0000","last_undegraded":"2024-09-18T01:53:25.630903+0000","last_fullsized":"2024-09-18T01:53:25.630903+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:51:35.915729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001186821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:21.988579+0000","last_change":"2024-09-18T01:16:24.045936+0000","last_active":"2024-09-18T01:53:21.988579+0000","last_peered":"2024-09-18T01:53:21.988579+0000","last_clean":"2024-09-18T01:53:21.988579+0000","last_became_active":"2024-09-18T01:16:22.048463+0000","last_became_peered":"2024-09-18T01:16:22.048463+0000","last_unstale":"2024-09-18T01:53:21.988579+0000","last_undegraded":"2024-09-18T01:53:21.988579+0000","last_fullsized":"2024-09-18T01:53:21.988579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:09:27.308691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050943500000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.881863+0000","last_change":"2024-09-18T01:16:24.060226+0000","last_active":"2024-09-18T01:53:25.881863+0000","last_peered":"2024-09-18T01:53:25.881863+0000","last_clean":"2024-09-18T01:53:25.881863+0000","last_became_active":"2024-09-18T01:16:22.048849+0000","last_became_peered":"2024-09-18T01:16:22.048849+0000","last_unstale":"2024-09-18T01:53:25.881863+0000","last_undegraded":"2024-09-18T01:53:25.881863+0000","last_fullsized":"2024-09-18T01:53:25.881863+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:32.872881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056915500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.881668+0000","last_change":"2024-09-18T01:16:24.060127+0000","last_active":"2024-09-18T01:53:25.881668+0000","last_peered":"2024-09-18T01:53:25.881668+0000","last_clean":"2024-09-18T01:53:25.881668+0000","last_became_active":"2024-09-18T01:16:22.045793+0000","last_became_peered":"2024-09-18T01:16:22.045793+0000","last_unstale":"2024-09-18T01:53:25.881668+0000","last_undegraded":"2024-09-18T01:53:25.881668+0000","last_fullsized":"2024-09-18T01:53:25.881668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:08:15.672511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048395300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.633940+0000","last_change":"2024-09-18T01:16:24.049520+0000","last_active":"2024-09-18T01:53:25.633940+0000","last_peered":"2024-09-18T01:53:25.633940+0000","last_clean":"2024-09-18T01:53:25.633940+0000","last_became_active":"2024-09-18T01:16:22.045810+0000","last_became_peered":"2024-09-18T01:16:22.045810+0000","last_unstale":"2024-09-18T01:53:25.633940+0000","last_undegraded":"2024-09-18T01:53:25.633940+0000","last_fullsized":"2024-09-18T01:53:25.633940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:33.391698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010551429999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1075,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.628762+0000","last_change":"2024-09-18T01:16:17.153205+0000","last_active":"2024-09-18T01:53:25.628762+0000","last_peered":"2024-09-18T01:53:25.628762+0000","last_clean":"2024-09-18T01:53:25.628762+0000","last_became_active":"2024-09-18T01:16:17.151939+0000","last_became_peered":"2024-09-18T01:16:17.151939+0000","last_unstale":"2024-09-18T01:53:25.628762+0000","last_undegraded":"2024-09-18T01:53:25.628762+0000","last_fullsized":"2024-09-18T01:53:25.628762+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_clean_scrub_stamp":"2024-09-18T01:16:16.017299+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:51:08.601096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706117,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151580,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220260,"statfs":{"total":96636764160,"available":96481546240,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706118,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165532,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206308,"statfs":{"total":96636764160,"available":96467259392,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133468,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94238372,"statfs":{"total":96636764160,"available":96500092928,"internally_reserved":0,"allocated":3874816,"data_stored":2071306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132774341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:51.169 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-18T01:53:51.460 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:53:51.460 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":471,"fsid":"5799928d-af76-40cf-a91d-c6362506fa2c","created":"2024-09-18T01:16:05.571663+0000","modified":"2024-09-18T01:53:21.982234+0000","last_up_change":"2024-09-18T01:16:14.978708+0000","last_in_change":"2024-09-18T01:16:06.473310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T01:16:16.014269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T01:16:20.752157+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"81a7cd46-8949-4110-bb1e-add42dfb2660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6809","nonce":3770637869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6811","nonce":3770637869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6815","nonce":3770637869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":3770637869},{"type":"v1","addr":"172.21.15.66:6813","nonce":3770637869}]},"public_addr":"172.21.15.66:6809/3770637869","cluster_addr":"172.21.15.66:6811/3770637869","heartbeat_back_addr":"172.21.15.66:6815/3770637869","heartbeat_front_addr":"172.21.15.66:6813/3770637869","state":["exists","up"]},{"osd":1,"uuid":"88aec54c-891e-4b4f-967d-f7bb2e7d4c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6817","nonce":2946278758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6819","nonce":2946278758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6823","nonce":2946278758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2946278758},{"type":"v1","addr":"172.21.15.66:6821","nonce":2946278758}]},"public_addr":"172.21.15.66:6817/2946278758","cluster_addr":"172.21.15.66:6819/2946278758","heartbeat_back_addr":"172.21.15.66:6823/2946278758","heartbeat_front_addr":"172.21.15.66:6821/2946278758","state":["exists","up"]},{"osd":2,"uuid":"2630c86f-e0ff-400e-93ca-84a26a630e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6801","nonce":1507114421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6803","nonce":1507114421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6807","nonce":1507114421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1507114421},{"type":"v1","addr":"172.21.15.66:6805","nonce":1507114421}]},"public_addr":"172.21.15.66:6801/1507114421","cluster_addr":"172.21.15.66:6803/1507114421","heartbeat_back_addr":"172.21.15.66:6807/1507114421","heartbeat_front_addr":"172.21.15.66:6805/1507114421","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:13.460411+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:13.004326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T01:16:12.762131+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":196,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T01:53:52.474 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-18T01:53:52.475 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T01:53:52.615 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-09-18T01:53:52.615 INFO:teuthology.orchestra.run.smithi066.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T01:53:52.615 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-09-18T01:53:52.626 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-18T01:53:52.917 INFO:teuthology.orchestra.run.smithi066.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T01:53:52.930 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-18T01:53:52.930 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T01:53:53.056 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-09-18T01:53:53.056 INFO:teuthology.orchestra.run.smithi066.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T01:53:53.057 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-09-18T01:53:53.067 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-18T01:53:53.357 INFO:teuthology.orchestra.run.smithi066.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T01:53:53.370 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-18T01:53:53.370 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T01:53:53.495 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-09-18T01:53:53.495 INFO:teuthology.orchestra.run.smithi066.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T01:53:53.495 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-09-18T01:53:53.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-18T01:53:53.796 INFO:teuthology.orchestra.run.smithi066.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T01:53:53.809 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T01:53:54.097 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:53:54.097 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-09-18T01:53:54.109 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-09-18T01:53:52.567244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":450580,"kb_used_data":11480,"kb_used_omap":4,"kb_used_meta":439035,"kb_avail":282664940,"statfs":{"total":289910292480,"available":289448898560,"internally_reserved":0,"allocated":11755520,"data_stored":7394692,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":449572175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.630324+0000","last_change":"2024-09-18T01:16:24.048908+0000","last_active":"2024-09-18T01:53:25.630324+0000","last_peered":"2024-09-18T01:53:25.630324+0000","last_clean":"2024-09-18T01:53:25.630324+0000","last_became_active":"2024-09-18T01:16:22.049384+0000","last_became_peered":"2024-09-18T01:16:22.049384+0000","last_unstale":"2024-09-18T01:53:25.630324+0000","last_undegraded":"2024-09-18T01:53:25.630324+0000","last_fullsized":"2024-09-18T01:53:25.630324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:08:44.387269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067731499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.634629+0000","last_change":"2024-09-18T01:16:24.049185+0000","last_active":"2024-09-18T01:53:25.634629+0000","last_peered":"2024-09-18T01:53:25.634629+0000","last_clean":"2024-09-18T01:53:25.634629+0000","last_became_active":"2024-09-18T01:16:22.048383+0000","last_became_peered":"2024-09-18T01:16:22.048383+0000","last_unstale":"2024-09-18T01:53:25.634629+0000","last_undegraded":"2024-09-18T01:53:25.634629+0000","last_fullsized":"2024-09-18T01:53:25.634629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:33:41.916036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076761799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.633082+0000","last_change":"2024-09-18T01:16:24.048791+0000","last_active":"2024-09-18T01:53:25.633082+0000","last_peered":"2024-09-18T01:53:25.633082+0000","last_clean":"2024-09-18T01:53:25.633082+0000","last_became_active":"2024-09-18T01:16:22.049251+0000","last_became_peered":"2024-09-18T01:16:22.049251+0000","last_unstale":"2024-09-18T01:53:25.633082+0000","last_undegraded":"2024-09-18T01:53:25.633082+0000","last_fullsized":"2024-09-18T01:53:25.633082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:25.091421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057751099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.630903+0000","last_change":"2024-09-18T01:16:24.049406+0000","last_active":"2024-09-18T01:53:25.630903+0000","last_peered":"2024-09-18T01:53:25.630903+0000","last_clean":"2024-09-18T01:53:25.630903+0000","last_became_active":"2024-09-18T01:16:22.049287+0000","last_became_peered":"2024-09-18T01:16:22.049287+0000","last_unstale":"2024-09-18T01:53:25.630903+0000","last_undegraded":"2024-09-18T01:53:25.630903+0000","last_fullsized":"2024-09-18T01:53:25.630903+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:51:35.915729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001186821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:21.988579+0000","last_change":"2024-09-18T01:16:24.045936+0000","last_active":"2024-09-18T01:53:21.988579+0000","last_peered":"2024-09-18T01:53:21.988579+0000","last_clean":"2024-09-18T01:53:21.988579+0000","last_became_active":"2024-09-18T01:16:22.048463+0000","last_became_peered":"2024-09-18T01:16:22.048463+0000","last_unstale":"2024-09-18T01:53:21.988579+0000","last_undegraded":"2024-09-18T01:53:21.988579+0000","last_fullsized":"2024-09-18T01:53:21.988579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:09:27.308691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050943500000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":946,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.881863+0000","last_change":"2024-09-18T01:16:24.060226+0000","last_active":"2024-09-18T01:53:25.881863+0000","last_peered":"2024-09-18T01:53:25.881863+0000","last_clean":"2024-09-18T01:53:25.881863+0000","last_became_active":"2024-09-18T01:16:22.048849+0000","last_became_peered":"2024-09-18T01:16:22.048849+0000","last_unstale":"2024-09-18T01:53:25.881863+0000","last_undegraded":"2024-09-18T01:53:25.881863+0000","last_fullsized":"2024-09-18T01:53:25.881863+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:32.872881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056915500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.881668+0000","last_change":"2024-09-18T01:16:24.060127+0000","last_active":"2024-09-18T01:53:25.881668+0000","last_peered":"2024-09-18T01:53:25.881668+0000","last_clean":"2024-09-18T01:53:25.881668+0000","last_became_active":"2024-09-18T01:16:22.045793+0000","last_became_peered":"2024-09-18T01:16:22.045793+0000","last_unstale":"2024-09-18T01:53:25.881668+0000","last_undegraded":"2024-09-18T01:53:25.881668+0000","last_fullsized":"2024-09-18T01:53:25.881668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:08:15.672511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048395300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":947,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.633940+0000","last_change":"2024-09-18T01:16:24.049520+0000","last_active":"2024-09-18T01:53:25.633940+0000","last_peered":"2024-09-18T01:53:25.633940+0000","last_clean":"2024-09-18T01:53:25.633940+0000","last_became_active":"2024-09-18T01:16:22.045810+0000","last_became_peered":"2024-09-18T01:16:22.045810+0000","last_unstale":"2024-09-18T01:53:25.633940+0000","last_undegraded":"2024-09-18T01:53:25.633940+0000","last_fullsized":"2024-09-18T01:53:25.633940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:21.024534+0000","last_clean_scrub_stamp":"2024-09-18T01:16:21.024534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:38:33.391698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010551429999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1075,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:25.628762+0000","last_change":"2024-09-18T01:16:17.153205+0000","last_active":"2024-09-18T01:53:25.628762+0000","last_peered":"2024-09-18T01:53:25.628762+0000","last_clean":"2024-09-18T01:53:25.628762+0000","last_became_active":"2024-09-18T01:16:17.151939+0000","last_became_peered":"2024-09-18T01:16:17.151939+0000","last_unstale":"2024-09-18T01:53:25.628762+0000","last_undegraded":"2024-09-18T01:53:25.628762+0000","last_fullsized":"2024-09-18T01:53:25.628762+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:16:16.017299+0000","last_clean_scrub_stamp":"2024-09-18T01:16:16.017299+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:51:08.601096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706118,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151580,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220260,"statfs":{"total":96636764160,"available":96481546240,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706119,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165532,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206308,"statfs":{"total":96636764160,"available":96467259392,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133468,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94238372,"statfs":{"total":96636764160,"available":96500092928,"internally_reserved":0,"allocated":3874816,"data_stored":2071306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132774341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:53:54.110 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-18T01:16:21.024534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=21, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.110 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-18T01:16:21.024534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=21, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.110 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-18T01:16:21.024534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=21, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.110 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T01:16:21.024534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=21, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.110 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-18T01:16:21.024534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=21, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.110 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-18T01:16:21.024534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=21, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.111 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-18T01:16:21.024534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=21, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.111 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-18T01:16:21.024534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=21, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.111 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T01:16:16.017299+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=16, tm_sec=16, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=1, tm_min=53, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T01:53:54.111 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T01:54:14.112 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T01:54:14.412 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-09-18T01:54:14.412 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-09-18T01:54:14.424 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":1608,"stamp":"2024-09-18T01:54:12.571515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":450568,"kb_used_data":11480,"kb_used_omap":4,"kb_used_meta":439035,"kb_avail":282664952,"statfs":{"total":289910292480,"available":289448910848,"internally_reserved":0,"allocated":11755520,"data_stored":7394692,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":449572175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002773"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":955,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:58.939545+0000","last_change":"2024-09-18T01:53:58.939572+0000","last_active":"2024-09-18T01:53:58.939545+0000","last_peered":"2024-09-18T01:53:58.939545+0000","last_clean":"2024-09-18T01:53:58.939545+0000","last_became_active":"2024-09-18T01:16:22.049384+0000","last_became_peered":"2024-09-18T01:16:22.049384+0000","last_unstale":"2024-09-18T01:53:58.939545+0000","last_undegraded":"2024-09-18T01:53:58.939545+0000","last_fullsized":"2024-09-18T01:53:58.939545+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:58.939430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:58.939430+0000","last_clean_scrub_stamp":"2024-09-18T01:53:58.939430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:41:28.662464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067731499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":955,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:55.859014+0000","last_change":"2024-09-18T01:53:55.859038+0000","last_active":"2024-09-18T01:53:55.859014+0000","last_peered":"2024-09-18T01:53:55.859014+0000","last_clean":"2024-09-18T01:53:55.859014+0000","last_became_active":"2024-09-18T01:16:22.048383+0000","last_became_peered":"2024-09-18T01:16:22.048383+0000","last_unstale":"2024-09-18T01:53:55.859014+0000","last_undegraded":"2024-09-18T01:53:55.859014+0000","last_fullsized":"2024-09-18T01:53:55.859014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:55.858908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:55.858908+0000","last_clean_scrub_stamp":"2024-09-18T01:53:55.858908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:57:35.669841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076761799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":956,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:56.878084+0000","last_change":"2024-09-18T01:53:56.878084+0000","last_active":"2024-09-18T01:53:56.878084+0000","last_peered":"2024-09-18T01:53:56.878084+0000","last_clean":"2024-09-18T01:53:56.878084+0000","last_became_active":"2024-09-18T01:16:22.049251+0000","last_became_peered":"2024-09-18T01:16:22.049251+0000","last_unstale":"2024-09-18T01:53:56.878084+0000","last_undegraded":"2024-09-18T01:53:56.878084+0000","last_fullsized":"2024-09-18T01:53:56.878084+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:56.877900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:56.877900+0000","last_clean_scrub_stamp":"2024-09-18T01:53:56.877900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:15:52.367911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057751099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":956,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:57.896755+0000","last_change":"2024-09-18T01:53:57.896780+0000","last_active":"2024-09-18T01:53:57.896755+0000","last_peered":"2024-09-18T01:53:57.896755+0000","last_clean":"2024-09-18T01:53:57.896755+0000","last_became_active":"2024-09-18T01:16:22.049287+0000","last_became_peered":"2024-09-18T01:16:22.049287+0000","last_unstale":"2024-09-18T01:53:57.896755+0000","last_undegraded":"2024-09-18T01:53:57.896755+0000","last_fullsized":"2024-09-18T01:53:57.896755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:57.896664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:57.896664+0000","last_clean_scrub_stamp":"2024-09-18T01:53:57.896664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:15:13.323343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001186821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"471'3","reported_seq":958,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:53.592504+0000","last_change":"2024-09-18T01:53:53.592504+0000","last_active":"2024-09-18T01:53:53.592504+0000","last_peered":"2024-09-18T01:53:53.592504+0000","last_clean":"2024-09-18T01:53:53.592504+0000","last_became_active":"2024-09-18T01:16:22.048463+0000","last_became_peered":"2024-09-18T01:16:22.048463+0000","last_unstale":"2024-09-18T01:53:53.592504+0000","last_undegraded":"2024-09-18T01:53:53.592504+0000","last_fullsized":"2024-09-18T01:53:53.592504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'3","last_scrub_stamp":"2024-09-18T01:53:53.592364+0000","last_deep_scrub":"471'3","last_deep_scrub_stamp":"2024-09-18T01:53:53.592364+0000","last_clean_scrub_stamp":"2024-09-18T01:53:53.592364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:46:38.279952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050943500000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":956,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:55.126270+0000","last_change":"2024-09-18T01:53:55.126270+0000","last_active":"2024-09-18T01:53:55.126270+0000","last_peered":"2024-09-18T01:53:55.126270+0000","last_clean":"2024-09-18T01:53:55.126270+0000","last_became_active":"2024-09-18T01:16:22.048849+0000","last_became_peered":"2024-09-18T01:16:22.048849+0000","last_unstale":"2024-09-18T01:53:55.126270+0000","last_undegraded":"2024-09-18T01:53:55.126270+0000","last_fullsized":"2024-09-18T01:53:55.126270+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:55.126143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:55.126143+0000","last_clean_scrub_stamp":"2024-09-18T01:53:55.126143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:07:29.670548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056915500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":956,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:54.140652+0000","last_change":"2024-09-18T01:53:54.140671+0000","last_active":"2024-09-18T01:53:54.140652+0000","last_peered":"2024-09-18T01:53:54.140652+0000","last_clean":"2024-09-18T01:53:54.140652+0000","last_became_active":"2024-09-18T01:16:22.045793+0000","last_became_peered":"2024-09-18T01:16:22.045793+0000","last_unstale":"2024-09-18T01:53:54.140652+0000","last_undegraded":"2024-09-18T01:53:54.140652+0000","last_fullsized":"2024-09-18T01:53:54.140652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T01:53:54.140570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T01:53:54.140570+0000","last_clean_scrub_stamp":"2024-09-18T01:53:54.140570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:06:24.356304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048395300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"471'2","reported_seq":958,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:54.813839+0000","last_change":"2024-09-18T01:53:54.813839+0000","last_active":"2024-09-18T01:53:54.813839+0000","last_peered":"2024-09-18T01:53:54.813839+0000","last_clean":"2024-09-18T01:53:54.813839+0000","last_became_active":"2024-09-18T01:16:22.045810+0000","last_became_peered":"2024-09-18T01:16:22.045810+0000","last_unstale":"2024-09-18T01:53:54.813839+0000","last_undegraded":"2024-09-18T01:53:54.813839+0000","last_fullsized":"2024-09-18T01:53:54.813839+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'2","last_scrub_stamp":"2024-09-18T01:53:54.813675+0000","last_deep_scrub":"471'2","last_deep_scrub_stamp":"2024-09-18T01:53:54.813675+0000","last_clean_scrub_stamp":"2024-09-18T01:53:54.813675+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:59:38.582752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010551429999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"471'99","reported_seq":1086,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-18T01:53:53.781162+0000","last_change":"2024-09-18T01:53:53.781210+0000","last_active":"2024-09-18T01:53:53.781162+0000","last_peered":"2024-09-18T01:53:53.781162+0000","last_clean":"2024-09-18T01:53:53.781162+0000","last_became_active":"2024-09-18T01:16:17.151939+0000","last_became_peered":"2024-09-18T01:16:17.151939+0000","last_unstale":"2024-09-18T01:53:53.781162+0000","last_undegraded":"2024-09-18T01:53:53.781162+0000","last_fullsized":"2024-09-18T01:53:53.781162+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'99","last_scrub_stamp":"2024-09-18T01:53:53.781090+0000","last_deep_scrub":"471'99","last_deep_scrub_stamp":"2024-09-18T01:53:53.781090+0000","last_clean_scrub_stamp":"2024-09-18T01:53:53.781090+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:44:36.816186+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59950,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706122,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151576,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220264,"statfs":{"total":96636764160,"available":96481550336,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706123,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165528,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206312,"statfs":{"total":96636764160,"available":96467263488,"internally_reserved":0,"allocated":3940352,"data_stored":2661693,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133464,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94238376,"statfs":{"total":96636764160,"available":96500097024,"internally_reserved":0,"allocated":3874816,"data_stored":2071306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132774341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29975,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T01:54:14.425 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-18T01:54:14.730 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T01:54:14.730 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T01:54:14.730 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-18T01:54:14.730 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:54:14.773 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-18T01:54:14.773 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-18T01:54:14.774 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:54:14.818 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-18T01:54:14.818 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-18T01:54:14.818 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:54:14.873 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-18T01:54:14.874 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T01:54:14.874 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-18T01:54:14.874 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:54:14.900 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-18T01:54:14.900 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T01:54:14.900 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-18T01:54:14.900 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T01:54:14.953 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-18T01:54:14.953 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T01:54:14.953 DEBUG:teuthology.orchestra.run.smithi066:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-18T01:54:15.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:54:15.021 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-18T01:54:15.225 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:54:15.225 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-18T01:54:15.297 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:54:15.297 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-18T01:54:15.355 INFO:tasks.ceph:Archiving mon data... 2024-09-18T01:54:15.356 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706/data/cluster1.mon.a.tgz 2024-09-18T01:54:15.356 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-09-18T01:54:15.376 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.Q2soc45ZMy 2024-09-18T01:54:15.376 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.Q2soc45ZMy 2024-09-18T01:54:15.529 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.Q2soc45ZMy 2024-09-18T01:54:15.614 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.Q2soc45ZMy is 368KB 2024-09-18T01:54:15.712 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.Q2soc45ZMy 2024-09-18T01:54:15.727 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T01:54:15.727 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T01:54:15.769 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T01:54:15.848 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-18T01:54:15.903 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-18T01:54:15.903 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T01:54:15.905 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706/remote/smithi066/crash 2024-09-18T01:54:15.906 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T01:54:15.931 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706/remote/smithi115/crash 2024-09-18T01:54:15.932 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T01:54:15.961 INFO:tasks.ceph:Compressing logs... 2024-09-18T01:54:15.961 DEBUG:teuthology.orchestra.run.smithi066:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T01:54:15.976 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32231.log 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35338.log 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.32231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36849.log 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32231.log.gz 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35338.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35338.log.gz 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35336.log 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.36849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36849.log.gz 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35096.log 2024-09-18T01:54:16.021 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35336.log.gz 2024-09-18T01:54:16.022 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36807.log 2024-09-18T01:54:16.022 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35096.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35096.log.gz 2024-09-18T01:54:16.022 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32132.log 2024-09-18T01:54:16.022 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.36807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-18T01:54:16.022 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36807.log.gz 2024-09-18T01:54:16.022 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.32132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37017.log 2024-09-18T01:54:16.022 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32132.log.gz 2024-09-18T01:54:16.022 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35464.log 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.37017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35760.log 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37017.log.gz 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35462.log 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35464.log.gz 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35760.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35760.log.gz 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35550.log 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37269.log 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35462.log.gz 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35550.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.admin.35550.log.gz -- 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr: /var/log/ceph/cluster1-osd.2.log 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.37269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32295.log 2024-09-18T01:54:16.023 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37269.log.gz 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37059.log 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.32295.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32295.log.gz 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35718.log 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.37059.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37059.log.gz 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36933.log 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35718.log.gz 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.36933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37309.log 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36933.log.gz 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37185.log 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.37309.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37309.log.gz 2024-09-18T01:54:16.024 INFO:teuthology.orchestra.run.smithi066.stderr: -- /var/log/ceph/cluster1-client.admin.31894.log 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.37185.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37185.log.gz 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32052.log 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.31894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36891.log 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31894.log.gz 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.32052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35214.log 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32052.log.gz 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.36891.log: /var/log/ceph/cluster1-client.admin.34873.log 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36891.log.gz 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35214.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35337.log 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35214.log.gz 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.34873.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34873.log.gz 2024-09-18T01:54:16.025 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35337.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35337.log.gz 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35172.log 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37143.log 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35172.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35256.log 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35172.log.gz 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.37143.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37143.log.gz 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35256.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.35256.log.gz 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27403.log 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35463.log 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph.tmp-client.admin.27403.log: /var/log/ceph/cluster1-osd.0.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27403.log.gz 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr: 93.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35589.log 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35463.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35463.log.gz 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35040.log 2024-09-18T01:54:16.026 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35589.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36975.log 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35589.log.gz 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35590.log 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35040.log: /var/log/ceph/cluster1-client.admin.36975.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35040.log.gz 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36975.log.gz 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37227.log 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.35590.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35676.log 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35590.log.gz 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37101.log 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.37227.log: /var/log/ceph/cluster1-client.admin.35676.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37227.log.gz 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35676.log.gz 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.37101.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35121.log 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37101.log.gz 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32092.log 2024-09-18T01:54:16.027 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.35121.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.35121.log.gz 2024-09-18T01:54:16.028 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cluster1-client.admin.32092.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32092.log.gz 2024-09-18T01:54:16.028 INFO:teuthology.orchestra.run.smithi066.stderr: 87.6% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-18T01:54:16.058 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51945.log 2024-09-18T01:54:16.058 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43896.log 2024-09-18T01:54:16.058 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70513.log 2024-09-18T01:54:16.059 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43376.log 2024-09-18T01:54:16.060 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61453.log 2024-09-18T01:54:16.060 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51945.log.gz 2024-09-18T01:54:16.060 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70513.log.gz 2024-09-18T01:54:16.060 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.43376.log: /var/log/ceph/cluster2-client.mirror.44406.log 2024-09-18T01:54:16.060 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53266.log 2024-09-18T01:54:16.060 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43376.log.gz 2024-09-18T01:54:16.060 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44406.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61453.log.gz 2024-09-18T01:54:16.060 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48971.log 2024-09-18T01:54:16.064 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40476.log 2024-09-18T01:54:16.064 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44406.log.gz 2024-09-18T01:54:16.064 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55980.log 2024-09-18T01:54:16.064 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53266.log.gz 2024-09-18T01:54:16.064 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56096.log 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr: 76.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40476.log.gz 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63478.log 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62034.log 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56096.log.gz 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60855.log 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58757.log 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63478.log.gz 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43896.log.gz 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62034.log.gz 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55980.log.gz 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr:gzip/var/log/ceph/cluster1-client.mirror.60855.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49426.log 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45555.log 2024-09-18T01:54:16.065 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58757.log.gz 2024-09-18T01:54:16.066 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68456.log 2024-09-18T01:54:16.066 INFO:teuthology.orchestra.run.smithi115.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49426.log.gz 2024-09-18T01:54:16.066 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52632.log 2024-09-18T01:54:16.066 INFO:teuthology.orchestra.run.smithi115.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45555.log.gz 2024-09-18T01:54:16.066 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.68456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49158.log 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59506.log 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60855.log.gz 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49158.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68456.log.gz 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44666.log 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49158.log.gz 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57930.log 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59506.log: /var/log/ceph/cluster2-client.mirror.44666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49447.log 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70716.log 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44666.log.gz 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59506.log.gz 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52632.log.gz 2024-09-18T01:54:16.083 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40933.log 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49447.log.gz 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60742.log 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57930.log.gz 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40933.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70716.log.gz 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58046.log 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37344.log 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60742.log.gz 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41045.log 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58046.log.gz 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56856.log 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.37344.log: /var/log/ceph/cluster1-client.mirror.41045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36709.log 2024-09-18T01:54:16.084 INFO:teuthology.orchestra.run.smithi115.stderr: 74.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41045.log.gz 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37344.log.gz 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56952.log 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56856.log.gz 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35459.log 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.0.36709.log: /var/log/ceph/cluster1-client.mirror.56952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50608.log 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35459.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56952.log.gz 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35459.log.gz 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63560.log 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45320.log 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50608.log: /var/log/ceph/cluster2-client.mirror.63560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40436.log 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65733.log 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:gzip/var/log/ceph/cluster2-client.mirror.40436.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.36585.log 2024-09-18T01:54:16.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65733.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45320.log.gz 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40846.log 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50608.log.gz 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44466.log 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36585.log.gz 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65733.log.gz 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40846.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63560.log.gz 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71415.log 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72290.log 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44466.log: /var/log/ceph/cluster2-client.mirror.71415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53613.log 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71415.log.gz 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.72290.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72290.log.gz 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44466.log.gz 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.37455.log 2024-09-18T01:54:16.086 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68829.log 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.37455.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53613.log.gz 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37455.log.gz 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50933.log 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65007.log 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68829.log: /var/log/ceph/cluster2-client.mirror.50933.log: gzip -5 --verbose 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68829.log.gz 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr: -- /var/log/ceph/cluster1-client.mirror.51832.log 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47884.log 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50933.log.gz 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51832.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65007.log.gz 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57745.log 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51832.log.gz 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70122.log 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47884.log: /var/log/ceph/cluster1-client.mirror.57745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49668.log 2024-09-18T01:54:16.087 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47884.log.gz 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40846.log.gz 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57745.log.gz 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60971.log 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63354.log 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49668.log: /var/log/ceph/cluster1-client.mirror.60971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71666.log 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70122.log.gz 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63354.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60971.log.gz 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54582.log 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63354.log.gz 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49668.log.gz/var/log/ceph/cluster1-client.mirror.71666.log: 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62083.log 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71666.log.gz 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37600.log 2024-09-18T01:54:16.088 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54582.log: /var/log/ceph/cluster1-client.mirror.62083.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45041.log 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62083.log.gz 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49076.log 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54582.log.gz 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46225.log 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45041.log: /var/log/ceph/cluster2-client.mirror.49076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44526.log 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49076.log.gz 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60814.log 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46225.log.gz 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44986.log 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57769.log 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60814.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44526.log.gz 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46649.log 2024-09-18T01:54:16.089 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60814.log.gz 2024-09-18T01:54:16.090 INFO:teuthology.orchestra.run.smithi115.stderr:gzip/var/log/ceph/cluster1-client.mirror.57769.log: 87.5% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.44986.log.gz -- 2024-09-18T01:54:16.090 INFO:teuthology.orchestra.run.smithi115.stderr: /var/log/ceph/cluster1-client.mirror.67862.log 2024-09-18T01:54:16.090 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46649.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57769.log.gz 2024-09-18T01:54:16.090 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62228.log 2024-09-18T01:54:16.108 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50102.log 2024-09-18T01:54:16.108 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67862.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67862.log.gz 2024-09-18T01:54:16.108 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70143.log 2024-09-18T01:54:16.108 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62228.log.gz 2024-09-18T01:54:16.108 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64242.log 2024-09-18T01:54:16.108 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44606.log 2024-09-18T01:54:16.108 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70143.log.gz 2024-09-18T01:54:16.108 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47864.log 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50102.log.gz 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44749.log 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64242.log: /var/log/ceph/cluster2-client.mirror.44606.log: /var/log/ceph/cluster2-client.mirror.47864.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44606.log.gz 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr: 58.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37600.log.gz 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr:gzip 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47864.log.gz 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47325.log 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59823.log 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47325.log: /var/log/ceph/cluster1-client.mirror.44749.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44749.log.gz 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36709.log.gz 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64242.log.gz 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36866.log 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr:gzip/var/log/ceph/cluster2-client.mirror.59823.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36459.log 2024-09-18T01:54:16.109 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.36866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63850.log 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.36459.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59935.log 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.36459.log.gz 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41454.log 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63850.log: /var/log/ceph/cluster2-client.mirror.59935.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59823.log.gz 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63850.log.gz 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.51025.log 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59935.log.gz 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.41454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42918.log 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41454.log.gz 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72265.log 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51025.log.gz 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69203.log 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.72265.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62782.log 2024-09-18T01:54:16.110 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.72265.log.gz 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52902.log 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69203.log: /var/log/ceph/cluster1-client.mirror.62782.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69203.log.gz 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62782.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63519.log 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52902.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52902.log.gz 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71576.log 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63684.log 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71576.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63519.log.gz 76.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71576.log.gz 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40349.log 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53456.log 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63684.log: /var/log/ceph/cluster1-client.mirror.40349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54855.log 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53456.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53456.log.gz 2024-09-18T01:54:16.111 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63684.log.gz 2024-09-18T01:54:16.112 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40805.log 2024-09-18T01:54:16.112 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54348.log 2024-09-18T01:54:16.112 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54855.log.gz 2024-09-18T01:54:16.112 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59484.log 2024-09-18T01:54:16.112 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54348.log: 88.0% 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54348.log.gz 2024-09-18T01:54:16.112 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40805.log.gz 2024-09-18T01:54:16.112 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38650.log 2024-09-18T01:54:16.114 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59484.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59484.log.gz 2024-09-18T01:54:16.115 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46995.log 2024-09-18T01:54:16.115 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.38650.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47580.log 2024-09-18T01:54:16.115 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46995.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46995.log.gz 2024-09-18T01:54:16.115 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35826.log 2024-09-18T01:54:16.116 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47580.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.38650.log.gz 2024-09-18T01:54:16.125 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57817.log 2024-09-18T01:54:16.125 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35826.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35826.log.gz 2024-09-18T01:54:16.136 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48413.log 2024-09-18T01:54:16.136 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57817.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40349.log.gz 2024-09-18T01:54:16.136 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57817.log.gz 2024-09-18T01:54:16.156 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60031.log 2024-09-18T01:54:16.170 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60561.log 2024-09-18T01:54:16.171 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60031.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60031.log.gz 2024-09-18T01:54:16.190 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44834.log 2024-09-18T01:54:16.191 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60561.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60561.log.gz 2024-09-18T01:54:16.199 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55590.log 2024-09-18T01:54:16.213 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44834.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44834.log.gz 2024-09-18T01:54:16.215 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66310.log 2024-09-18T01:54:16.215 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55590.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55590.log.gz 2024-09-18T01:54:16.222 INFO:teuthology.orchestra.run.smithi066.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-18T01:54:16.234 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40372.log 2024-09-18T01:54:16.235 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66310.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66310.log.gz 2024-09-18T01:54:16.243 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35913.log 2024-09-18T01:54:16.265 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62830.log 2024-09-18T01:54:16.265 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35913.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35913.log.gz 2024-09-18T01:54:16.275 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36866.log.gz 2024-09-18T01:54:16.281 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40372.log.gz 2024-09-18T01:54:16.285 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53432.log 2024-09-18T01:54:16.286 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62830.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62830.log.gz 2024-09-18T01:54:16.296 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47443.log 2024-09-18T01:54:16.298 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53432.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53432.log.gz 2024-09-18T01:54:16.318 INFO:teuthology.orchestra.run.smithi066.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-18T01:54:16.322 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44626.log 2024-09-18T01:54:16.323 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47443.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47443.log.gz 2024-09-18T01:54:16.331 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63271.log 2024-09-18T01:54:16.340 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59346.log 2024-09-18T01:54:16.344 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63271.log: 89.4% 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44626.log.gz 2024-09-18T01:54:16.354 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63271.log.gz 2024-09-18T01:54:16.354 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63457.log 2024-09-18T01:54:16.355 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59346.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59346.log.gz 2024-09-18T01:54:16.363 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49298.log 2024-09-18T01:54:16.372 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57793.log 2024-09-18T01:54:16.376 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49298.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63457.log.gz 2024-09-18T01:54:16.391 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41428.log 2024-09-18T01:54:16.392 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57793.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57793.log.gz 2024-09-18T01:54:16.404 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42378.log 2024-09-18T01:54:16.413 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52106.log 2024-09-18T01:54:16.427 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48354.log 2024-09-18T01:54:16.429 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52106.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52106.log.gz 2024-09-18T01:54:16.431 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42378.log.gz 2024-09-18T01:54:16.445 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53890.log 2024-09-18T01:54:16.445 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48354.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48354.log.gz 2024-09-18T01:54:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67885.log 2024-09-18T01:54:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53890.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53890.log.gz 2024-09-18T01:54:16.478 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68222.log 2024-09-18T01:54:16.478 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67885.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67885.log.gz 2024-09-18T01:54:16.492 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45423.log 2024-09-18T01:54:16.493 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68222.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68222.log.gz 2024-09-18T01:54:16.515 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50724.log 2024-09-18T01:54:16.516 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45423.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45423.log.gz 2024-09-18T01:54:16.525 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58091.log 2024-09-18T01:54:16.525 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50724.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50724.log.gz 2024-09-18T01:54:16.539 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49712.log 2024-09-18T01:54:16.551 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58091.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48450.log 2024-09-18T01:54:16.552 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49712.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49712.log.gz 2024-09-18T01:54:16.554 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58091.log.gz 2024-09-18T01:54:16.564 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54095.log 2024-09-18T01:54:16.565 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48450.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48450.log.gz 2024-09-18T01:54:16.569 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48971.log.gz 2024-09-18T01:54:16.580 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68087.log 2024-09-18T01:54:16.580 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54095.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54095.log.gz 2024-09-18T01:54:16.594 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54215.log 2024-09-18T01:54:16.594 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68087.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68087.log.gz 2024-09-18T01:54:16.605 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46165.log 2024-09-18T01:54:16.605 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54215.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54215.log.gz 2024-09-18T01:54:16.616 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41775.log 2024-09-18T01:54:16.632 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59706.log 2024-09-18T01:54:16.632 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41775.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41775.log.gz 2024-09-18T01:54:16.646 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55108.log 2024-09-18T01:54:16.646 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59706.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59706.log.gz 2024-09-18T01:54:16.649 INFO:teuthology.orchestra.run.smithi115.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47580.log.gz 2024-09-18T01:54:16.663 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46851.log 2024-09-18T01:54:16.664 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55108.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55108.log.gz 2024-09-18T01:54:16.672 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61573.log 2024-09-18T01:54:16.681 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53107.log 2024-09-18T01:54:16.684 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61573.log: 87.5% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46851.log.gz 2024-09-18T01:54:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45099.log 2024-09-18T01:54:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53107.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61573.log.gz 2024-09-18T01:54:16.706 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.69580.log 2024-09-18T01:54:16.707 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45099.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45099.log.gz 2024-09-18T01:54:16.707 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53107.log.gz 2024-09-18T01:54:16.726 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70607.log 2024-09-18T01:54:16.729 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.0.69580.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.69580.log.gz 2024-09-18T01:54:16.740 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40709.log 2024-09-18T01:54:16.740 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70607.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70607.log.gz 2024-09-18T01:54:16.740 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48413.log.gz 2024-09-18T01:54:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49135.log 2024-09-18T01:54:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40709.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40709.log.gz 2024-09-18T01:54:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41384.log 2024-09-18T01:54:16.772 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60371.log 2024-09-18T01:54:16.785 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41384.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44546.log 2024-09-18T01:54:16.786 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60371.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60371.log.gz 2024-09-18T01:54:16.789 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41384.log.gz 2024-09-18T01:54:16.801 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38361.log 2024-09-18T01:54:16.802 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44546.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44546.log.gz 2024-09-18T01:54:16.807 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45041.log.gz 2024-09-18T01:54:16.807 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57540.log 2024-09-18T01:54:16.808 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.38361.log: 76.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38361.log.gz 2024-09-18T01:54:16.816 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52266.log 2024-09-18T01:54:16.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57540.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57540.log.gz 2024-09-18T01:54:16.833 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70084.log 2024-09-18T01:54:16.843 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52266.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50194.log 2024-09-18T01:54:16.847 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70084.log: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52266.log.gz 2024-09-18T01:54:16.855 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40933.log.gz 2024-09-18T01:54:16.857 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70084.log.gz 2024-09-18T01:54:16.868 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56832.log 2024-09-18T01:54:16.868 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50194.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50194.log.gz 2024-09-18T01:54:16.883 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42535.log 2024-09-18T01:54:16.883 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56832.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56832.log.gz 2024-09-18T01:54:16.891 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63374.log 2024-09-18T01:54:16.895 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42535.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42535.log.gz 2024-09-18T01:54:16.900 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39422.log 2024-09-18T01:54:16.911 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68498.log 2024-09-18T01:54:16.911 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39422.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47325.log.gz 2024-09-18T01:54:16.912 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55749.log 2024-09-18T01:54:16.916 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.68498.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63374.log.gz 2024-09-18T01:54:16.921 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57263.log 2024-09-18T01:54:16.922 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55749.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55749.log.gz 2024-09-18T01:54:16.934 INFO:teuthology.orchestra.run.smithi115.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42918.log.gz 2024-09-18T01:54:16.934 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39422.log.gz 2024-09-18T01:54:16.934 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72610.log 2024-09-18T01:54:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57263.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57263.log.gz 2024-09-18T01:54:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72315.log 2024-09-18T01:54:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72610.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72610.log.gz 2024-09-18T01:54:16.938 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63622.log 2024-09-18T01:54:16.938 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.72315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72315.log.gz 2024-09-18T01:54:16.938 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63932.log 2024-09-18T01:54:16.941 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63622.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63622.log.gz 2024-09-18T01:54:16.941 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41321.log 2024-09-18T01:54:16.943 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63932.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63932.log.gz 2024-09-18T01:54:16.944 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72568.log 2024-09-18T01:54:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41321.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41321.log.gz 2024-09-18T01:54:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38716.log 2024-09-18T01:54:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72568.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72568.log.gz 2024-09-18T01:54:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44240.log 2024-09-18T01:54:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.38716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.36944.log 2024-09-18T01:54:16.952 INFO:teuthology.orchestra.run.smithi115.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38716.log.gz 2024-09-18T01:54:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.44240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61272.log 2024-09-18T01:54:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.2.36944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62348.log 2024-09-18T01:54:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61272.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61272.log.gz 2024-09-18T01:54:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54927.log 2024-09-18T01:54:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62348.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62348.log.gz 2024-09-18T01:54:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57585.log 2024-09-18T01:54:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54927.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54927.log.gz 2024-09-18T01:54:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71710.log 2024-09-18T01:54:16.969 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43352.log 2024-09-18T01:54:16.969 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71710.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46649.log.gz 2024-09-18T01:54:16.969 INFO:teuthology.orchestra.run.smithi115.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71710.log.gz 2024-09-18T01:54:16.969 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61130.log 2024-09-18T01:54:16.969 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69142.log 2024-09-18T01:54:16.969 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61130.log: /var/log/ceph/cluster2-client.mirror.43352.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43352.log.gz 2024-09-18T01:54:16.969 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44240.log.gz 2024-09-18T01:54:16.970 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57585.log.gz 2024-09-18T01:54:16.970 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58733.log 2024-09-18T01:54:16.970 INFO:teuthology.orchestra.run.smithi115.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61130.log.gz 2024-09-18T01:54:16.970 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69952.log 2024-09-18T01:54:16.970 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58733.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58733.log.gz 2024-09-18T01:54:16.970 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65069.log 2024-09-18T01:54:16.972 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69952.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69952.log.gz 2024-09-18T01:54:16.972 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39105.log 2024-09-18T01:54:16.972 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65069.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65069.log.gz 2024-09-18T01:54:16.972 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56812.log 2024-09-18T01:54:16.973 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.39105.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54396.log 2024-09-18T01:54:16.973 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56812.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56812.log.gz 2024-09-18T01:54:16.975 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59915.log 2024-09-18T01:54:16.975 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54396.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54396.log.gz 2024-09-18T01:54:16.976 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68065.log 2024-09-18T01:54:16.978 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59915.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59915.log.gz 2024-09-18T01:54:16.978 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61019.log 2024-09-18T01:54:16.978 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68065.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68065.log.gz 2024-09-18T01:54:16.978 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56144.log 2024-09-18T01:54:16.978 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61019.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61019.log.gz 2024-09-18T01:54:16.980 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52654.log 2024-09-18T01:54:16.980 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39105.log.gz 2024-09-18T01:54:16.980 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56144.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56144.log.gz 2024-09-18T01:54:16.980 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63209.log 2024-09-18T01:54:16.982 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52654.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52654.log.gz 2024-09-18T01:54:16.982 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54328.log 2024-09-18T01:54:16.982 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63209.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63209.log.gz 2024-09-18T01:54:16.982 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48311.log 2024-09-18T01:54:16.986 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54328.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54328.log.gz 2024-09-18T01:54:16.986 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41507.log 2024-09-18T01:54:16.986 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48311.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48311.log.gz 2024-09-18T01:54:16.986 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48131.log 2024-09-18T01:54:16.988 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41507.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41507.log.gz 2024-09-18T01:54:16.988 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47988.log 2024-09-18T01:54:16.989 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48131.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48131.log.gz 2024-09-18T01:54:16.989 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70064.log 2024-09-18T01:54:16.990 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47988.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.36944.log.gz 2024-09-18T01:54:17.002 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58227.log 2024-09-18T01:54:17.003 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70064.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70064.log.gz 2024-09-18T01:54:17.009 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38672.log 2024-09-18T01:54:17.009 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58227.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58227.log.gz 2024-09-18T01:54:17.022 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50082.log 2024-09-18T01:54:17.024 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.38672.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38672.log.gz 2024-09-18T01:54:17.033 INFO:teuthology.orchestra.run.smithi066.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-18T01:54:17.035 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37211.log 2024-09-18T01:54:17.037 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50082.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50082.log.gz 2024-09-18T01:54:17.048 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49298.log.gz 2024-09-18T01:54:17.048 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64346.log 2024-09-18T01:54:17.048 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.37211.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37211.log.gz 2024-09-18T01:54:17.048 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60118.log 2024-09-18T01:54:17.048 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64346.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64346.log.gz 2024-09-18T01:54:17.050 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69388.log 2024-09-18T01:54:17.050 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60118.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60118.log.gz 2024-09-18T01:54:17.050 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71127.log 2024-09-18T01:54:17.053 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69388.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69388.log.gz 2024-09-18T01:54:17.053 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46268.log 2024-09-18T01:54:17.053 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71127.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71127.log.gz 2024-09-18T01:54:17.053 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49217.log 2024-09-18T01:54:17.054 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46268.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46268.log.gz 2024-09-18T01:54:17.054 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70628.log 2024-09-18T01:54:17.066 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55313.log 2024-09-18T01:54:17.068 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70628.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70628.log.gz 2024-09-18T01:54:17.082 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67795.log 2024-09-18T01:54:17.082 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55313.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55313.log.gz 2024-09-18T01:54:17.083 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64470.log 2024-09-18T01:54:17.083 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67795.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67795.log.gz 2024-09-18T01:54:17.096 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50013.log 2024-09-18T01:54:17.098 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64470.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64470.log.gz 2024-09-18T01:54:17.098 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70757.log 2024-09-18T01:54:17.107 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50013.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41863.log 2024-09-18T01:54:17.107 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.50013.log.gz 2024-09-18T01:54:17.108 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70757.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70757.log.gz 2024-09-18T01:54:17.114 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67727.log 2024-09-18T01:54:17.123 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41863.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73028.log 2024-09-18T01:54:17.123 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67727.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67727.log.gz 2024-09-18T01:54:17.132 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41863.log.gz 2024-09-18T01:54:17.132 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51302.log 2024-09-18T01:54:17.145 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.73028.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73028.log.gz 2024-09-18T01:54:17.146 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43982.log 2024-09-18T01:54:17.147 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51302.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51302.log.gz 2024-09-18T01:54:17.156 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43201.log 2024-09-18T01:54:17.171 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39990.log 2024-09-18T01:54:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43201.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43982.log.gz 2024-09-18T01:54:17.177 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43201.log.gz 2024-09-18T01:54:17.185 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43853.log 2024-09-18T01:54:17.186 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.39990.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39990.log.gz 2024-09-18T01:54:17.204 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58251.log 2024-09-18T01:54:17.213 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43853.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59210.log 2024-09-18T01:54:17.222 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58251.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50630.log 2024-09-18T01:54:17.222 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59210.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43853.log.gz 2024-09-18T01:54:17.226 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58251.log.gz 2024-09-18T01:54:17.227 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41428.log.gz 2024-09-18T01:54:17.228 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41008.log 2024-09-18T01:54:17.230 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50630.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50630.log.gz 2024-09-18T01:54:17.230 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56374.log 2024-09-18T01:54:17.232 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41008.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41008.log.gz 2024-09-18T01:54:17.232 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45340.log 2024-09-18T01:54:17.233 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56374.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56374.log.gz 2024-09-18T01:54:17.234 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37481.log 2024-09-18T01:54:17.235 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45340.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45340.log.gz 2024-09-18T01:54:17.235 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65296.log 2024-09-18T01:54:17.236 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.37481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55960.log 2024-09-18T01:54:17.236 INFO:teuthology.orchestra.run.smithi115.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.37481.log.gz 2024-09-18T01:54:17.237 INFO:teuthology.orchestra.run.smithi115.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59210.log.gz 2024-09-18T01:54:17.239 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65296.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65296.log.gz 2024-09-18T01:54:17.239 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39820.log 2024-09-18T01:54:17.242 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55960.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55960.log.gz 2024-09-18T01:54:17.242 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56467.log 2024-09-18T01:54:17.243 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71244.log 2024-09-18T01:54:17.245 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56467.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56467.log.gz 2024-09-18T01:54:17.245 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41727.log 2024-09-18T01:54:17.247 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71244.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71244.log.gz 2024-09-18T01:54:17.247 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62938.log 2024-09-18T01:54:17.248 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41727.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41727.log.gz 2024-09-18T01:54:17.248 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56326.log 2024-09-18T01:54:17.251 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62938.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62938.log.gz 2024-09-18T01:54:17.251 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48949.log 2024-09-18T01:54:17.253 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56326.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56326.log.gz 2024-09-18T01:54:17.253 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68887.log 2024-09-18T01:54:17.254 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48949.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48949.log.gz 2024-09-18T01:54:17.254 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65172.log 2024-09-18T01:54:17.255 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68887.log: 84.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68887.log.gz 2024-09-18T01:54:17.255 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69310.log 2024-09-18T01:54:17.255 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39820.log.gz 2024-09-18T01:54:17.258 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65172.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65172.log.gz 2024-09-18T01:54:17.259 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67975.log 2024-09-18T01:54:17.260 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69310.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69310.log.gz 2024-09-18T01:54:17.260 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54119.log 2024-09-18T01:54:17.261 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67975.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67975.log.gz 2024-09-18T01:54:17.261 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57374.log 2024-09-18T01:54:17.262 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54119.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54119.log.gz 2024-09-18T01:54:17.262 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65503.log 2024-09-18T01:54:17.264 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57374.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57374.log.gz 2024-09-18T01:54:17.264 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67457.log 2024-09-18T01:54:17.266 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65503.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65503.log.gz 2024-09-18T01:54:17.266 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52612.log 2024-09-18T01:54:17.267 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67457.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67457.log.gz 2024-09-18T01:54:17.267 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58111.log 2024-09-18T01:54:17.269 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52612.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52612.log.gz 2024-09-18T01:54:17.270 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.61646.log 2024-09-18T01:54:17.273 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58111.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58111.log.gz 2024-09-18T01:54:17.273 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42400.log 2024-09-18T01:54:17.281 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.3.61646.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40436.log.gz 2024-09-18T01:54:17.281 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67030.log 2024-09-18T01:54:17.285 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42400.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42400.log.gz 2024-09-18T01:54:17.285 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66647.log 2024-09-18T01:54:17.285 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67030.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67030.log.gz 2024-09-18T01:54:17.285 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62107.log 2024-09-18T01:54:17.285 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66647.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66647.log.gz 2024-09-18T01:54:17.287 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55707.log 2024-09-18T01:54:17.287 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62107.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62107.log.gz 2024-09-18T01:54:17.287 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46060.log 2024-09-18T01:54:17.289 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55707.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55707.log.gz 2024-09-18T01:54:17.289 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42790.log 2024-09-18T01:54:17.290 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46060.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46060.log.gz 2024-09-18T01:54:17.290 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64759.log 2024-09-18T01:54:17.292 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42790.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42790.log.gz 2024-09-18T01:54:17.292 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47384.log 2024-09-18T01:54:17.293 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64759.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64759.log.gz 2024-09-18T01:54:17.293 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42831.log 2024-09-18T01:54:17.294 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47384.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47384.log.gz 2024-09-18T01:54:17.295 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58639.log 2024-09-18T01:54:17.301 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42831.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42831.log.gz 2024-09-18T01:54:17.301 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70476.log 2024-09-18T01:54:17.303 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58639.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58639.log.gz 2024-09-18T01:54:17.303 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58552.log 2024-09-18T01:54:17.304 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70476.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70476.log.gz 2024-09-18T01:54:17.304 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48610.log 2024-09-18T01:54:17.305 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58552.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58552.log.gz 2024-09-18T01:54:17.306 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59464.log 2024-09-18T01:54:17.307 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48610.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48610.log.gz 2024-09-18T01:54:17.307 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69083.log 2024-09-18T01:54:17.310 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59464.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59464.log.gz 2024-09-18T01:54:17.310 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69735.log 2024-09-18T01:54:17.311 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69083.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69083.log.gz 2024-09-18T01:54:17.311 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66760.log 2024-09-18T01:54:17.314 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69735.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69735.log.gz 2024-09-18T01:54:17.314 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71077.log 2024-09-18T01:54:17.314 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66760.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66760.log.gz 2024-09-18T01:54:17.314 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63953.log 2024-09-18T01:54:17.317 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71077.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.61646.log.gz 2024-09-18T01:54:17.317 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53519.log 2024-09-18T01:54:17.318 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63953.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63953.log.gz 2024-09-18T01:54:17.319 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50588.log 2024-09-18T01:54:17.320 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53519.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49217.log.gz 2024-09-18T01:54:17.320 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53519.log.gz 2024-09-18T01:54:17.320 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62170.log 2024-09-18T01:54:17.321 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.50588.log: -- /var/log/ceph/cluster2-client.mirror.63974.log 2024-09-18T01:54:17.323 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62170.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62170.log.gz 2024-09-18T01:54:17.323 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50588.log.gz 2024-09-18T01:54:17.323 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48173.log 2024-09-18T01:54:17.323 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56579.log 2024-09-18T01:54:17.324 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48173.log: /var/log/ceph/cluster2-client.mirror.63974.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63974.log.gz 2024-09-18T01:54:17.324 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38907.log 2024-09-18T01:54:17.324 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56579.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56579.log.gz 2024-09-18T01:54:17.325 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53750.log 2024-09-18T01:54:17.325 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.38907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71855.log 2024-09-18T01:54:17.328 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53750.log: gzip -5 --verbose 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53750.log.gz 2024-09-18T01:54:17.329 INFO:teuthology.orchestra.run.smithi115.stderr: -- /var/log/ceph/cluster2-client.mirror.71751.log 2024-09-18T01:54:17.330 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71855.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71855.log.gz 2024-09-18T01:54:17.330 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67142.log 2024-09-18T01:54:17.332 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71751.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38907.log.gz 2024-09-18T01:54:17.333 INFO:teuthology.orchestra.run.smithi115.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71751.log.gz 2024-09-18T01:54:17.333 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50377.log 2024-09-18T01:54:17.334 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67142.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67142.log.gz 2024-09-18T01:54:17.335 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44772.log 2024-09-18T01:54:17.336 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50377.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50377.log.gz 2024-09-18T01:54:17.336 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61180.log 2024-09-18T01:54:17.337 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.44772.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44772.log.gz 2024-09-18T01:54:17.338 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45121.log 2024-09-18T01:54:17.339 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61180.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61180.log.gz 2024-09-18T01:54:17.339 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36375.log 2024-09-18T01:54:17.346 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45121.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46165.log.gz 2024-09-18T01:54:17.348 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45996.log 2024-09-18T01:54:17.348 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.36375.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36375.log.gz 2024-09-18T01:54:17.348 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47202.log 2024-09-18T01:54:17.362 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46752.log 2024-09-18T01:54:17.363 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47202.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47202.log.gz 2024-09-18T01:54:17.375 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56192.log 2024-09-18T01:54:17.375 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46752.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46752.log.gz 2024-09-18T01:54:17.389 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54698.log 2024-09-18T01:54:17.389 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56192.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56192.log.gz 2024-09-18T01:54:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59230.log 2024-09-18T01:54:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54698.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54698.log.gz 2024-09-18T01:54:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71035.log 2024-09-18T01:54:17.414 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59230.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59230.log.gz 2024-09-18T01:54:17.425 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49135.log.gz 2024-09-18T01:54:17.426 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.49505.log 2024-09-18T01:54:17.428 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71035.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71035.log.gz 2024-09-18T01:54:17.428 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41252.log 2024-09-18T01:54:17.432 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.0.49505.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.49505.log.gz 2024-09-18T01:54:17.432 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53477.log 2024-09-18T01:54:17.434 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41252.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41252.log.gz 2024-09-18T01:54:17.434 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69693.log 2024-09-18T01:54:17.435 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53477.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53477.log.gz 2024-09-18T01:54:17.436 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69931.log 2024-09-18T01:54:17.437 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69693.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69693.log.gz 2024-09-18T01:54:17.438 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.71576.log 2024-09-18T01:54:17.438 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69931.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69931.log.gz 2024-09-18T01:54:17.439 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57880.log 2024-09-18T01:54:17.441 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.1.71576.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.71576.log.gz 2024-09-18T01:54:17.442 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52878.log 2024-09-18T01:54:17.443 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57880.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57880.log.gz 2024-09-18T01:54:17.443 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63436.log 2024-09-18T01:54:17.445 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52878.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52878.log.gz 2024-09-18T01:54:17.445 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56740.log 2024-09-18T01:54:17.448 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63436.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63436.log.gz 2024-09-18T01:54:17.448 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.36966.log 2024-09-18T01:54:17.450 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56740.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56740.log.gz 2024-09-18T01:54:17.450 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57352.log 2024-09-18T01:54:17.451 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.3.36966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57444.log 2024-09-18T01:54:17.453 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57352.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57352.log.gz 2024-09-18T01:54:17.454 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54650.log 2024-09-18T01:54:17.456 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57444.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57444.log.gz 2024-09-18T01:54:17.456 INFO:teuthology.orchestra.run.smithi115.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69142.log.gz 2024-09-18T01:54:17.456 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72526.log 2024-09-18T01:54:17.456 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.54650.log: --verbose -- /var/log/ceph/cluster1-client.mirror.67930.log 2024-09-18T01:54:17.464 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54650.log.gz/var/log/ceph/cluster2-client.admin.72526.log: 2024-09-18T01:54:17.464 INFO:teuthology.orchestra.run.smithi115.stderr: 25.9% -- replaced with /var/log/ceph/cluster2-client.admin.72526.log.gz 2024-09-18T01:54:17.464 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47303.log 2024-09-18T01:54:17.464 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.67930.log: /var/log/ceph/cluster1-client.admin.36291.log 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67930.log.gz 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44814.log 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.36291.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36291.log.gz 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47303.log.gz 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56306.log 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66625.log 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56306.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44814.log.gz 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42445.log 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66625.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56306.log.gz 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37673.log 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.66625.log.gz 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65836.log 2024-09-18T01:54:17.465 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.42445.log: /var/log/ceph/cluster2-client.mirror.37673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57468.log 2024-09-18T01:54:17.466 INFO:teuthology.orchestra.run.smithi115.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.42445.log.gz 2024-09-18T01:54:17.466 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58480.log 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37673.log.gz 2024-09-18T01:54:17.466 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:17.466 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65836.log: /var/log/ceph/cluster1-client.mirror.57468.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57468.log.gz 2024-09-18T01:54:17.466 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67367.log 2024-09-18T01:54:17.469 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58480.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58480.log.gz 2024-09-18T01:54:17.469 INFO:teuthology.orchestra.run.smithi115.stderr:gzip 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65836.log.gz 2024-09-18T01:54:17.469 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47803.log 2024-09-18T01:54:17.469 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55474.log 2024-09-18T01:54:17.469 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47803.log: /var/log/ceph/cluster1-client.mirror.67367.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67367.log.gz 2024-09-18T01:54:17.469 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47803.log.gz 2024-09-18T01:54:17.469 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50427.log 2024-09-18T01:54:17.470 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63023.log 2024-09-18T01:54:17.470 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55474.log: /var/log/ceph/cluster2-client.mirror.50427.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50427.log.gz 2024-09-18T01:54:17.470 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58203.log 2024-09-18T01:54:17.470 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55474.log.gz 2024-09-18T01:54:17.470 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38976.log 2024-09-18T01:54:17.472 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58203.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58203.log.gz 2024-09-18T01:54:17.472 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60995.log 2024-09-18T01:54:17.472 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.38976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58133.log 2024-09-18T01:54:17.472 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60995.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60995.log.gz 2024-09-18T01:54:17.473 INFO:teuthology.orchestra.run.smithi115.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63023.log.gz 2024-09-18T01:54:17.483 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38976.log.gz 2024-09-18T01:54:17.487 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.36966.log.gz 2024-09-18T01:54:17.488 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51094.log 2024-09-18T01:54:17.488 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64573.log 2024-09-18T01:54:17.489 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51094.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58133.log.gz 2024-09-18T01:54:17.490 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66692.log 2024-09-18T01:54:17.492 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64573.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64573.log.gz 2024-09-18T01:54:17.492 INFO:teuthology.orchestra.run.smithi115.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51094.log.gz 2024-09-18T01:54:17.493 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38883.log 2024-09-18T01:54:17.493 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51873.log 2024-09-18T01:54:17.493 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66692.log.gz 2024-09-18T01:54:17.493 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.38883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35699.log 2024-09-18T01:54:17.493 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51989.log 2024-09-18T01:54:17.493 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38883.log.gz 2024-09-18T01:54:17.493 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35699.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35699.log.gz 2024-09-18T01:54:17.494 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45885.log 2024-09-18T01:54:17.494 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51989.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51989.log.gz 2024-09-18T01:54:17.494 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51873.log.gz 2024-09-18T01:54:17.494 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45572.log 2024-09-18T01:54:17.494 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49736.log 2024-09-18T01:54:17.497 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.45885.log: /var/log/ceph/cluster2-client.mirror.45572.log: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45885.log.gz 2024-09-18T01:54:17.497 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64056.log 2024-09-18T01:54:17.497 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49736.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45572.log.gz 2024-09-18T01:54:17.497 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49736.log.gz 2024-09-18T01:54:17.497 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58070.log 2024-09-18T01:54:17.497 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45362.log 2024-09-18T01:54:17.498 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58070.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58070.log.gz 2024-09-18T01:54:17.498 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42054.log 2024-09-18T01:54:17.499 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45362.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64056.log.gz 2024-09-18T01:54:17.501 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37566.log 2024-09-18T01:54:17.501 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.42054.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42054.log.gz 2024-09-18T01:54:17.501 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44111.log 2024-09-18T01:54:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51001.log 2024-09-18T01:54:17.502 INFO:teuthology.orchestra.run.smithi115.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37566.log.gz 2024-09-18T01:54:17.508 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.44111.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44111.log.gz 2024-09-18T01:54:17.508 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57065.log 2024-09-18T01:54:17.510 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51001.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51001.log.gz 2024-09-18T01:54:17.510 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.71937.log 2024-09-18T01:54:17.511 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57065.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57065.log.gz 2024-09-18T01:54:17.512 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63001.log 2024-09-18T01:54:17.514 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.1.71937.log: 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.71937.log.gz 2024-09-18T01:54:17.514 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46188.log 2024-09-18T01:54:17.517 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63001.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63001.log.gz 2024-09-18T01:54:17.517 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.41045.log 2024-09-18T01:54:17.518 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46188.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46188.log.gz 2024-09-18T01:54:17.518 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62517.log 2024-09-18T01:54:17.521 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.1.41045.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.41045.log.gz 2024-09-18T01:54:17.521 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42174.log 2024-09-18T01:54:17.522 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62517.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62517.log.gz 2024-09-18T01:54:17.522 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52704.log 2024-09-18T01:54:17.523 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42174.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42174.log.gz 2024-09-18T01:54:17.524 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50174.log 2024-09-18T01:54:17.525 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52704.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52704.log.gz 2024-09-18T01:54:17.525 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66445.log 2024-09-18T01:54:17.527 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50174.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50174.log.gz 2024-09-18T01:54:17.527 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71332.log 2024-09-18T01:54:17.528 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66445.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66445.log.gz 2024-09-18T01:54:17.529 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55361.log 2024-09-18T01:54:17.530 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71332.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71332.log.gz 2024-09-18T01:54:17.530 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55156.log 2024-09-18T01:54:17.531 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55361.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55361.log.gz 2024-09-18T01:54:17.531 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41905.log 2024-09-18T01:54:17.532 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55156.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55156.log.gz 2024-09-18T01:54:17.532 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51483.log 2024-09-18T01:54:17.534 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41905.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41905.log.gz 2024-09-18T01:54:17.534 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57516.log 2024-09-18T01:54:17.535 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51483.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51483.log.gz 2024-09-18T01:54:17.535 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69000.log 2024-09-18T01:54:17.537 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57516.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57516.log.gz 2024-09-18T01:54:17.537 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64366.log 2024-09-18T01:54:17.538 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69000.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69000.log.gz 2024-09-18T01:54:17.538 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41300.log 2024-09-18T01:54:17.541 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64366.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64366.log.gz 2024-09-18T01:54:17.541 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72071.log 2024-09-18T01:54:17.542 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41300.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41300.log.gz 2024-09-18T01:54:17.543 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38769.log 2024-09-18T01:54:17.544 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.72071.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72071.log.gz 2024-09-18T01:54:17.544 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53224.log 2024-09-18T01:54:17.545 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.38769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56675.log 2024-09-18T01:54:17.547 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53224.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53224.log.gz 2024-09-18T01:54:17.547 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66782.log 2024-09-18T01:54:17.549 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56675.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56675.log.gz 2024-09-18T01:54:17.549 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49829.log 2024-09-18T01:54:17.550 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66782.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66782.log.gz 2024-09-18T01:54:17.550 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36669.log 2024-09-18T01:54:17.552 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49829.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49829.log.gz 2024-09-18T01:54:17.554 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46566.log 2024-09-18T01:54:17.554 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36669.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36669.log.gz 2024-09-18T01:54:17.554 INFO:teuthology.orchestra.run.smithi115.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38769.log.gz 2024-09-18T01:54:17.554 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55201.log 2024-09-18T01:54:17.563 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46566.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68498.log.gz 2024-09-18T01:54:17.564 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58386.log 2024-09-18T01:54:17.567 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55201.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55201.log.gz 2024-09-18T01:54:17.567 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32420.log 2024-09-18T01:54:17.568 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58386.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58386.log.gz 2024-09-18T01:54:17.569 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43962.log 2024-09-18T01:54:17.569 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.32420.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32420.log.gz 2024-09-18T01:54:17.569 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47825.log 2024-09-18T01:54:17.576 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43962.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43962.log.gz 2024-09-18T01:54:17.576 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50290.log 2024-09-18T01:54:17.592 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63829.log 2024-09-18T01:54:17.593 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50290.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50290.log.gz 2024-09-18T01:54:17.595 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47988.log.gz 2024-09-18T01:54:17.596 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53336.log 2024-09-18T01:54:17.597 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63829.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63829.log.gz 2024-09-18T01:54:17.597 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54444.log 2024-09-18T01:54:17.598 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53336.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53336.log.gz 2024-09-18T01:54:17.598 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48025.log 2024-09-18T01:54:17.600 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54444.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54444.log.gz 2024-09-18T01:54:17.600 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47137.log 2024-09-18T01:54:17.600 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48025.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48025.log.gz 2024-09-18T01:54:17.601 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44306.log 2024-09-18T01:54:17.601 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47137.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47137.log.gz 2024-09-18T01:54:17.602 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50495.log 2024-09-18T01:54:17.604 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44306.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44306.log.gz 2024-09-18T01:54:17.604 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69268.log 2024-09-18T01:54:17.604 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50495.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50495.log.gz 2024-09-18T01:54:17.605 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59189.log 2024-09-18T01:54:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69268.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69268.log.gz 2024-09-18T01:54:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38694.log 2024-09-18T01:54:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59189.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59189.log.gz 2024-09-18T01:54:17.608 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69776.log 2024-09-18T01:54:17.609 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.38694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61361.log 2024-09-18T01:54:17.610 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69776.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69776.log.gz 2024-09-18T01:54:17.610 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67660.log 2024-09-18T01:54:17.612 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61361.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61361.log.gz 2024-09-18T01:54:17.612 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44068.log 2024-09-18T01:54:17.614 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67660.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67660.log.gz 2024-09-18T01:54:17.614 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52991.log 2024-09-18T01:54:17.621 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.44068.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44068.log.gz 2024-09-18T01:54:17.622 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42335.log 2024-09-18T01:54:17.622 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52991.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38694.log.gz 2024-09-18T01:54:17.623 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52991.log.gz 2024-09-18T01:54:17.623 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71474.log 2024-09-18T01:54:17.624 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42335.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45121.log.gz 2024-09-18T01:54:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44134.log 2024-09-18T01:54:17.628 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71474.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42335.log.gz 2024-09-18T01:54:17.629 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49321.log 2024-09-18T01:54:17.633 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44134.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45996.log.gz 2024-09-18T01:54:17.635 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65276.log 2024-09-18T01:54:17.635 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49321.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49321.log.gz 2024-09-18T01:54:17.636 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61928.log 2024-09-18T01:54:17.636 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65276.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44134.log.gz 2024-09-18T01:54:17.636 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53983.log 2024-09-18T01:54:17.636 INFO:teuthology.orchestra.run.smithi115.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65276.log.gz 2024-09-18T01:54:17.636 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63581.log 2024-09-18T01:54:17.637 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53983.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61928.log.gz 2024-09-18T01:54:17.637 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53914.log 2024-09-18T01:54:17.639 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63581.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63581.log.gz 2024-09-18T01:54:17.639 INFO:teuthology.orchestra.run.smithi115.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53983.log.gz 2024-09-18T01:54:17.639 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49688.log 2024-09-18T01:54:17.639 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55891.log 2024-09-18T01:54:17.639 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53914.log.gz 2024-09-18T01:54:17.641 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60421.log 2024-09-18T01:54:17.641 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55891.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49688.log.gz 2024-09-18T01:54:17.641 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53155.log 2024-09-18T01:54:17.641 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55891.log.gz 2024-09-18T01:54:17.641 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52198.log 2024-09-18T01:54:17.641 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53155.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53155.log.gz 2024-09-18T01:54:17.643 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60421.log.gz 2024-09-18T01:54:17.644 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40284.log 2024-09-18T01:54:17.644 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.52198.log: --verbose -- /var/log/ceph/cluster2-client.mirror.64490.log 2024-09-18T01:54:17.644 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49849.log 2024-09-18T01:54:17.644 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52198.log.gz 2024-09-18T01:54:17.644 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64325.log 2024-09-18T01:54:17.646 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49849.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64490.log.gz 2024-09-18T01:54:17.647 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49849.log.gz 2024-09-18T01:54:17.647 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61752.log 2024-09-18T01:54:17.647 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55686.log 2024-09-18T01:54:17.647 INFO:teuthology.orchestra.run.smithi115.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40284.log.gz 2024-09-18T01:54:17.659 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48173.log.gz 2024-09-18T01:54:17.659 INFO:teuthology.orchestra.run.smithi115.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64325.log.gz 2024-09-18T01:54:17.659 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61752.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61752.log.gz 2024-09-18T01:54:17.659 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48470.log 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52218.log 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48470.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55686.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72904.log 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48470.log.gz 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67052.log 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52218.log: /var/log/ceph/cluster2-client.admin.72904.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72904.log.gz 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52218.log.gz 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62871.log 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49175.log 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67052.log.gz 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44326.log 2024-09-18T01:54:17.660 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44386.log 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44326.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49175.log.gz 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62871.log.gz 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44326.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64738.log 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43286.log 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54674.log 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44386.log.gz 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48891.log 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54674.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54674.log.gz 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43286.log.gz 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66580.log 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48891.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64738.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70349.log 2024-09-18T01:54:17.661 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66580.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66580.log.gz 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45260.log 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63126.log 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45260.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45260.log.gz 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 91.2% --verbose -- /var/log/ceph/cluster1-client.mirror.58830.log 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70349.log.gz 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44856.log 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63126.log: /var/log/ceph/cluster1-client.mirror.58830.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58830.log.gz 2024-09-18T01:54:17.662 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64697.log 2024-09-18T01:54:17.663 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44856.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63126.log.gz 2024-09-18T01:54:17.663 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49505.log 2024-09-18T01:54:17.664 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64697.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64697.log.gz 2024-09-18T01:54:17.664 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40308.log 2024-09-18T01:54:17.664 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49505.log: 81.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49505.log.gz 2024-09-18T01:54:17.664 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49093.log 2024-09-18T01:54:17.664 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39486.log 2024-09-18T01:54:17.664 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40308.log.gz 2024-09-18T01:54:17.665 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49093.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49093.log.gz 2024-09-18T01:54:17.666 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51620.log 2024-09-18T01:54:17.667 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72165.log 2024-09-18T01:54:17.669 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51620.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51620.log.gz 2024-09-18T01:54:17.669 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70997.log 2024-09-18T01:54:17.670 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.72165.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72165.log.gz 2024-09-18T01:54:17.670 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51136.log 2024-09-18T01:54:17.672 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70997.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70997.log.gz 2024-09-18T01:54:17.673 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45831.log 2024-09-18T01:54:17.674 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51136.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51136.log.gz 2024-09-18T01:54:17.674 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63643.log 2024-09-18T01:54:17.675 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45831.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45831.log.gz 2024-09-18T01:54:17.675 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50037.log 2024-09-18T01:54:17.676 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63643.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63643.log.gz 2024-09-18T01:54:17.677 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72092.log 2024-09-18T01:54:17.677 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50037.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50037.log.gz 2024-09-18T01:54:17.678 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51230.log 2024-09-18T01:54:17.678 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.72092.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72092.log.gz 2024-09-18T01:54:17.679 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.57174.log 2024-09-18T01:54:17.679 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51230.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51230.log.gz 2024-09-18T01:54:17.680 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65358.log 2024-09-18T01:54:17.689 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.0.57174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44506.log 2024-09-18T01:54:17.691 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65358.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65358.log.gz 2024-09-18T01:54:17.705 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61886.log 2024-09-18T01:54:17.706 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44506.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44506.log.gz 2024-09-18T01:54:17.706 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51073.log 2024-09-18T01:54:17.708 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61886.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61886.log.gz 2024-09-18T01:54:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36501.log 2024-09-18T01:54:17.723 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51073.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51073.log.gz 2024-09-18T01:54:17.734 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46347.log 2024-09-18T01:54:17.735 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36501.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36501.log.gz 2024-09-18T01:54:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37515.log 2024-09-18T01:54:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46347.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46347.log.gz 2024-09-18T01:54:17.762 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60602.log 2024-09-18T01:54:17.763 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.37515.log: 67.4% -- replaced with /var/log/ceph/cluster2-client.admin.37515.log.gz 2024-09-18T01:54:17.777 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44893.log 2024-09-18T01:54:17.779 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60602.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60602.log.gz 2024-09-18T01:54:17.789 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72820.log 2024-09-18T01:54:17.790 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44893.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44893.log.gz 2024-09-18T01:54:17.798 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66467.log 2024-09-18T01:54:17.807 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41204.log 2024-09-18T01:54:17.807 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66467.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66467.log.gz 2024-09-18T01:54:17.811 INFO:teuthology.orchestra.run.smithi115.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72820.log.gz 2024-09-18T01:54:17.826 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69973.log 2024-09-18T01:54:17.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41204.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41204.log.gz 2024-09-18T01:54:17.845 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46566.log.gz 2024-09-18T01:54:17.857 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36165.log 2024-09-18T01:54:17.858 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69973.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69973.log.gz 2024-09-18T01:54:17.875 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39238.log 2024-09-18T01:54:17.875 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36165.log.gz 2024-09-18T01:54:17.887 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59322.log 2024-09-18T01:54:17.900 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45202.log 2024-09-18T01:54:17.900 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59322.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59322.log.gz 2024-09-18T01:54:17.901 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39238.log.gz 2024-09-18T01:54:17.909 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43132.log 2024-09-18T01:54:17.912 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45202.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48891.log.gz 2024-09-18T01:54:17.921 INFO:teuthology.orchestra.run.smithi115.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.57174.log.gz 2024-09-18T01:54:17.921 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39334.log 2024-09-18T01:54:17.922 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43132.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43132.log.gz 2024-09-18T01:54:17.925 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43060.log 2024-09-18T01:54:17.925 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39334.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52148.log 2024-09-18T01:54:17.925 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43060.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43060.log.gz 2024-09-18T01:54:17.925 INFO:teuthology.orchestra.run.smithi115.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39334.log.gz 2024-09-18T01:54:17.937 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46627.log 2024-09-18T01:54:17.938 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52148.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52148.log.gz 2024-09-18T01:54:17.938 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61966.log 2024-09-18T01:54:17.939 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46627.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46627.log.gz 2024-09-18T01:54:17.939 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67750.log 2024-09-18T01:54:17.941 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61966.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61966.log.gz 2024-09-18T01:54:17.941 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36884.log 2024-09-18T01:54:17.942 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67750.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67750.log.gz 2024-09-18T01:54:17.942 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59802.log 2024-09-18T01:54:17.943 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.0.36884.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58851.log 2024-09-18T01:54:17.945 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59802.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59802.log.gz 2024-09-18T01:54:17.945 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67952.log 2024-09-18T01:54:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35619.log 2024-09-18T01:54:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67952.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67952.log.gz 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58851.log.gz 2024-09-18T01:54:17.949 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69522.log 2024-09-18T01:54:17.949 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35619.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35619.log.gz 2024-09-18T01:54:17.949 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58781.log 2024-09-18T01:54:17.950 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69522.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69522.log.gz 2024-09-18T01:54:17.950 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46732.log 2024-09-18T01:54:17.951 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58781.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58781.log.gz 2024-09-18T01:54:17.951 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46444.log 2024-09-18T01:54:17.951 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46732.log: 2024-09-18T01:54:17.952 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64077.log 2024-09-18T01:54:17.953 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46444.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46444.log.gz 2024-09-18T01:54:17.953 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39286.log 2024-09-18T01:54:17.954 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64077.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64077.log.gz 2024-09-18T01:54:17.957 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70105.log 2024-09-18T01:54:17.957 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69368.log 2024-09-18T01:54:17.957 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70105.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70105.log.gz 2024-09-18T01:54:17.957 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67345.log 2024-09-18T01:54:17.958 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69368.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69368.log.gz 2024-09-18T01:54:17.959 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37310.log 2024-09-18T01:54:17.959 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67345.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67345.log.gz 2024-09-18T01:54:17.960 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63664.log 2024-09-18T01:54:17.960 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.37310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56559.log 2024-09-18T01:54:17.960 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63664.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63664.log.gz 2024-09-18T01:54:17.961 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62758.log 2024-09-18T01:54:17.962 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56559.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56559.log.gz 2024-09-18T01:54:17.963 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69062.log 2024-09-18T01:54:17.963 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62758.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62758.log.gz 2024-09-18T01:54:17.964 INFO:teuthology.orchestra.run.smithi115.stderr: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37310.log.gz 2024-09-18T01:54:17.966 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39286.log.gz 2024-09-18T01:54:17.966 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67592.log 2024-09-18T01:54:17.971 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69062.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69062.log.gz 2024-09-18T01:54:17.971 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48811.log 2024-09-18T01:54:17.971 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67592.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67592.log.gz 2024-09-18T01:54:17.972 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68245.log 2024-09-18T01:54:17.986 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53179.log 2024-09-18T01:54:17.986 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68245.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68245.log.gz 2024-09-18T01:54:17.995 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48811.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.36884.log.gz 2024-09-18T01:54:17.995 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65420.log 2024-09-18T01:54:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53179.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53179.log.gz 2024-09-18T01:54:18.012 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51642.log 2024-09-18T01:54:18.014 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65420.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65420.log.gz 2024-09-18T01:54:18.021 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65878.log 2024-09-18T01:54:18.033 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-18T01:54:18.034 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65878.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65878.log.gz 2024-09-18T01:54:18.034 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51642.log.gz 2024-09-18T01:54:18.042 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48211.log 2024-09-18T01:54:18.045 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-osd.2.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47825.log.gz 2024-09-18T01:54:18.046 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58436.log 2024-09-18T01:54:18.050 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48211.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48211.log.gz 2024-09-18T01:54:18.050 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41481.log 2024-09-18T01:54:18.052 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58436.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58436.log.gz 2024-09-18T01:54:18.053 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45445.log 2024-09-18T01:54:18.053 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.41481.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41481.log.gz 2024-09-18T01:54:18.054 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42246.log 2024-09-18T01:54:18.067 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64139.log 2024-09-18T01:54:18.068 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42246.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42246.log.gz 2024-09-18T01:54:18.081 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41840.log 2024-09-18T01:54:18.082 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64139.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64139.log.gz 2024-09-18T01:54:18.083 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46040.log 2024-09-18T01:54:18.097 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41840.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41840.log.gz 2024-09-18T01:54:18.099 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45614.log 2024-09-18T01:54:18.099 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46040.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46040.log.gz 2024-09-18T01:54:18.103 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71474.log.gz 2024-09-18T01:54:18.105 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52591.log 2024-09-18T01:54:18.105 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45614.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71077.log.gz 2024-09-18T01:54:18.114 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45180.log 2024-09-18T01:54:18.118 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52591.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52591.log.gz 2024-09-18T01:54:18.118 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52085.log 2024-09-18T01:54:18.126 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45180.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45180.log.gz 2024-09-18T01:54:18.126 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61224.log 2024-09-18T01:54:18.127 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52085.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52085.log.gz 2024-09-18T01:54:18.129 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57564.log 2024-09-18T01:54:18.129 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61224.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61224.log.gz 2024-09-18T01:54:18.129 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64842.log 2024-09-18T01:54:18.138 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67390.log 2024-09-18T01:54:18.147 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64842.log: 88.2% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57564.log.gz 2024-09-18T01:54:18.147 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64842.log.gz 2024-09-18T01:54:18.149 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63994.log 2024-09-18T01:54:18.149 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67390.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67390.log.gz 2024-09-18T01:54:18.149 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68110.log 2024-09-18T01:54:18.151 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63994.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63994.log.gz 2024-09-18T01:54:18.151 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64594.log 2024-09-18T01:54:18.153 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68110.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68110.log.gz 2024-09-18T01:54:18.154 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44930.log 2024-09-18T01:54:18.154 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64594.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64594.log.gz 2024-09-18T01:54:18.156 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45079.log 2024-09-18T01:54:18.156 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44930.log: 85.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44930.log.gz 2024-09-18T01:54:18.156 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51254.log 2024-09-18T01:54:18.157 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45079.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45079.log.gz 2024-09-18T01:54:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39540.log 2024-09-18T01:54:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51254.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51254.log.gz 2024-09-18T01:54:18.166 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43919.log 2024-09-18T01:54:18.168 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.39540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68850.log 2024-09-18T01:54:18.173 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43919.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43919.log.gz 2024-09-18T01:54:18.174 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65816.log 2024-09-18T01:54:18.178 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.68850.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68850.log.gz 2024-09-18T01:54:18.179 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64800.log 2024-09-18T01:54:18.179 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.39540.log.gz 2024-09-18T01:54:18.180 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65816.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65816.log.gz 2024-09-18T01:54:18.180 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52519.log 2024-09-18T01:54:18.182 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64800.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64800.log.gz 2024-09-18T01:54:18.184 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65110.log 2024-09-18T01:54:18.184 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52519.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52519.log.gz 2024-09-18T01:54:18.184 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71772.log 2024-09-18T01:54:18.192 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65110.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65110.log.gz 2024-09-18T01:54:18.194 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62469.log 2024-09-18T01:54:18.194 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71772.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71772.log.gz 2024-09-18T01:54:18.194 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51760.log 2024-09-18T01:54:18.195 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62469.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62469.log.gz 2024-09-18T01:54:18.196 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70870.log 2024-09-18T01:54:18.196 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51760.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51760.log.gz 2024-09-18T01:54:18.202 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59754.log 2024-09-18T01:54:18.203 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70870.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70870.log.gz 2024-09-18T01:54:18.205 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60465.log 2024-09-18T01:54:18.206 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59754.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59754.log.gz 2024-09-18T01:54:18.206 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56651.log 2024-09-18T01:54:18.207 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60465.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60465.log.gz 2024-09-18T01:54:18.208 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46710.log 2024-09-18T01:54:18.209 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56651.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56651.log.gz 2024-09-18T01:54:18.209 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67502.log 2024-09-18T01:54:18.210 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46710.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46710.log.gz 2024-09-18T01:54:18.210 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40236.log 2024-09-18T01:54:18.211 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67502.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67502.log.gz 2024-09-18T01:54:18.211 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41927.log 2024-09-18T01:54:18.217 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68312.log 2024-09-18T01:54:18.217 INFO:teuthology.orchestra.run.smithi115.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40236.log.gz 2024-09-18T01:54:18.223 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41927.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45362.log.gz 2024-09-18T01:54:18.225 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40685.log 2024-09-18T01:54:18.225 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68312.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68312.log.gz 2024-09-18T01:54:18.225 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63602.log 2024-09-18T01:54:18.227 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40685.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40685.log.gz 2024-09-18T01:54:18.227 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47364.log 2024-09-18T01:54:18.227 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63602.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63602.log.gz 2024-09-18T01:54:18.228 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51579.log 2024-09-18T01:54:18.229 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47364.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47364.log.gz 2024-09-18T01:54:18.229 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49113.log 2024-09-18T01:54:18.230 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51579.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51579.log.gz 2024-09-18T01:54:18.232 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57424.log 2024-09-18T01:54:18.234 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49113.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49113.log.gz 2024-09-18T01:54:18.234 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50266.log 2024-09-18T01:54:18.234 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57424.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57424.log.gz 2024-09-18T01:54:18.234 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46106.log 2024-09-18T01:54:18.234 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50266.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50266.log.gz 2024-09-18T01:54:18.236 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41820.log 2024-09-18T01:54:18.238 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46106.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46106.log.gz 2024-09-18T01:54:18.239 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36543.log 2024-09-18T01:54:18.252 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67637.log 2024-09-18T01:54:18.252 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36543.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36543.log.gz 2024-09-18T01:54:18.256 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41820.log.gz 2024-09-18T01:54:18.259 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58963.log 2024-09-18T01:54:18.259 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67637.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67637.log.gz 2024-09-18T01:54:18.268 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46931.log 2024-09-18T01:54:18.269 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58963.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58963.log.gz 2024-09-18T01:54:18.277 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43833.log 2024-09-18T01:54:18.286 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48652.log 2024-09-18T01:54:18.290 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43833.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46931.log.gz 2024-09-18T01:54:18.293 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44856.log.gz 2024-09-18T01:54:18.294 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43552.log 2024-09-18T01:54:18.304 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48652.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43833.log.gz 2024-09-18T01:54:18.316 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50680.log 2024-09-18T01:54:18.321 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43552.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43552.log.gz 2024-09-18T01:54:18.322 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60624.log 2024-09-18T01:54:18.324 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50680.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50680.log.gz 2024-09-18T01:54:18.324 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69410.log 2024-09-18T01:54:18.325 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60624.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60624.log.gz 2024-09-18T01:54:18.325 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59983.log 2024-09-18T01:54:18.335 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48371.log 2024-09-18T01:54:18.344 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59983.log: 87.4% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46732.log.gz 2024-09-18T01:54:18.347 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53360.log 2024-09-18T01:54:18.348 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48371.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48371.log.gz 2024-09-18T01:54:18.348 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59983.log.gz 2024-09-18T01:54:18.357 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57998.log 2024-09-18T01:54:18.357 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53360.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53360.log.gz 2024-09-18T01:54:18.361 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49808.log 2024-09-18T01:54:18.361 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57998.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57998.log.gz 2024-09-18T01:54:18.362 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43108.log 2024-09-18T01:54:18.362 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49808.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49808.log.gz 2024-09-18T01:54:18.365 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64036.log 2024-09-18T01:54:18.365 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43108.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43108.log.gz 2024-09-18T01:54:18.365 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58871.log 2024-09-18T01:54:18.365 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64036.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64036.log.gz 2024-09-18T01:54:18.366 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68380.log 2024-09-18T01:54:18.374 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58871.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58871.log.gz 2024-09-18T01:54:18.374 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70022.log 2024-09-18T01:54:18.375 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68380.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68380.log.gz 2024-09-18T01:54:18.376 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42941.log 2024-09-18T01:54:18.376 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70022.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70022.log.gz 2024-09-18T01:54:18.376 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62369.log 2024-09-18T01:54:18.389 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50355.log 2024-09-18T01:54:18.391 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62369.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62369.log.gz 2024-09-18T01:54:18.396 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60766.log 2024-09-18T01:54:18.401 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50355.log: /var/log/ceph/cluster1-client.admin.42941.log: 88.2% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45202.log.gz 2024-09-18T01:54:18.402 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38361.log 2024-09-18T01:54:18.402 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60766.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60766.log.gz 2024-09-18T01:54:18.404 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53938.log 2024-09-18T01:54:18.404 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.1.38361.log: -- replaced with /var/log/ceph/cluster1-client.admin.42941.log.gz 2024-09-18T01:54:18.404 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55454.log 2024-09-18T01:54:18.404 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53938.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53938.log.gz 2024-09-18T01:54:18.405 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46205.log 2024-09-18T01:54:18.406 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55454.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55454.log.gz 2024-09-18T01:54:18.407 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65214.log 2024-09-18T01:54:18.408 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46205.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46205.log.gz 2024-09-18T01:54:18.408 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69755.log 2024-09-18T01:54:18.409 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65214.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65214.log.gz 2024-09-18T01:54:18.410 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65193.log 2024-09-18T01:54:18.410 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69755.log: -- replaced with /var/log/ceph/cluster1-client.mirror.50355.log.gz 2024-09-18T01:54:18.411 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44366.log 2024-09-18T01:54:18.412 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65193.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69755.log.gz 2024-09-18T01:54:18.412 INFO:teuthology.orchestra.run.smithi115.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65193.log.gz 2024-09-18T01:54:18.412 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47766.log 2024-09-18T01:54:18.417 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66110.log 2024-09-18T01:54:18.417 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47766.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47766.log.gz 2024-09-18T01:54:18.417 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44366.log.gz 2024-09-18T01:54:18.417 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46364.log 2024-09-18T01:54:18.417 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66110.log: gzip -5 --verbose -- 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66110.log.gz 2024-09-18T01:54:18.418 INFO:teuthology.orchestra.run.smithi115.stderr: /var/log/ceph/cluster2-client.mirror.63808.log 2024-09-18T01:54:18.418 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66377.log 2024-09-18T01:54:18.418 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63808.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46364.log.gz 2024-09-18T01:54:18.418 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58893.log 2024-09-18T01:54:18.418 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66377.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66377.log.gz 2024-09-18T01:54:18.418 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55867.log 2024-09-18T01:54:18.418 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58893.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63808.log.gz 2024-09-18T01:54:18.421 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58893.log.gz 2024-09-18T01:54:18.421 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64614.log 2024-09-18T01:54:18.421 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55867.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55867.log.gz 2024-09-18T01:54:18.421 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53013.log 2024-09-18T01:54:18.421 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64614.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65960.log 2024-09-18T01:54:18.422 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53013.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64614.log.gz 2024-09-18T01:54:18.422 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57838.log 2024-09-18T01:54:18.425 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65960.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65960.log.gz 2024-09-18T01:54:18.426 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36884.log 2024-09-18T01:54:18.427 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57838.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57838.log.gz 2024-09-18T01:54:18.428 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40869.log 2024-09-18T01:54:18.428 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.36884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52126.log 2024-09-18T01:54:18.429 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40869.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53013.log.gz 2024-09-18T01:54:18.430 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60349.log 2024-09-18T01:54:18.434 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52126.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40869.log.gz 2024-09-18T01:54:18.436 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67840.log 2024-09-18T01:54:18.439 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60349.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60349.log.gz 2024-09-18T01:54:18.440 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45445.log.gz 2024-09-18T01:54:18.440 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70043.log 2024-09-18T01:54:18.440 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3%/var/log/ceph/cluster1-client.mirror.67840.log: -- replaced with /var/log/ceph/cluster1-client.mirror.52126.log.gz 2024-09-18T01:54:18.440 INFO:teuthology.orchestra.run.smithi115.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67840.log.gz 2024-09-18T01:54:18.440 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.71127.log 2024-09-18T01:54:18.440 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57015.log 2024-09-18T01:54:18.440 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56993.log 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70043.log.gz 2024-09-18T01:54:18.440 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:18.441 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57015.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57015.log.gz 2024-09-18T01:54:18.441 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36207.log 2024-09-18T01:54:18.451 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61841.log 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36207.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36207.log.gz 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.0.71127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57950.log 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3%/var/log/ceph/cluster1-client.mirror.61841.log: -- replaced with /var/log/ceph/cluster1-client.mirror.56993.log.gz 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61841.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60927.log 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55496.log 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57950.log: /var/log/ceph/cluster2-client.mirror.60927.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57950.log.gz 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60927.log.gz 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr: -- /var/log/ceph/cluster2-client.mirror.65692.log 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.71127.log.gz 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54992.log 2024-09-18T01:54:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55496.log: /var/log/ceph/cluster2-client.mirror.65692.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65692.log.gz 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55496.log.gz 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63188.log 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71309.log 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70165.log 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71309.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71309.log.gz 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54992.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64284.log 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48789.log 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70165.log: /var/log/ceph/cluster2-client.mirror.64284.log: 90.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72190.log 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.48789.log: -- replaced with /var/log/ceph/cluster2-client.mirror.64284.log.gz 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63188.log.gz 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37634.log 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48811.log.gz 2024-09-18T01:54:18.453 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48789.log.gz 2024-09-18T01:54:18.454 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38361.log.gz 2024-09-18T01:54:18.454 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.72190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72190.log.gz 2024-09-18T01:54:18.457 INFO:teuthology.orchestra.run.smithi115.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.36884.log.gz 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71452.log 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69100.log 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37634.log.gz 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51186.log 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56350.log 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71452.log.gz 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39907.log 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69100.log.gz 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56350.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51186.log.gz 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56350.log.gz 2024-09-18T01:54:18.461 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36039.log 2024-09-18T01:54:18.462 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39018.log 2024-09-18T01:54:18.470 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5/var/log/ceph/cluster2-client.admin.36039.log: --verbose -- /var/log/ceph/cluster1-client.mirror.51507.log 0.0% 2024-09-18T01:54:18.470 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.36039.log.gz 2024-09-18T01:54:18.470 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67097.log 2024-09-18T01:54:18.470 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56446.log 2024-09-18T01:54:18.470 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51507.log.gz 2024-09-18T01:54:18.470 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67097.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39018.log.gz 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67097.log.gz 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66715.log 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56446.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51347.log 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66715.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56446.log.gz 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60877.log 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66715.log.gz 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62806.log 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51347.log: /var/log/ceph/cluster2-client.mirror.60877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54191.log 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62806.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62806.log.gz 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60877.log.gz 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67480.log 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54191.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51347.log.gz 2024-09-18T01:54:18.471 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54191.log.gz 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68806.log 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61646.log 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68806.log: /var/log/ceph/cluster1-client.mirror.67480.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67480.log.gz 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68806.log.gz 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59165.log 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr:gzip/var/log/ceph/cluster2-client.mirror.61646.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59543.log 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67210.log 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59543.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59543.log.gz 2024-09-18T01:54:18.472 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47558.log 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67210.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59165.log.gz 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67210.log.gz 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64676.log 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63498.log 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47558.log.gz 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51692.log 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63498.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61646.log.gz 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64676.log.gz 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61341.log 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63498.log.gz 2024-09-18T01:54:18.476 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65234.log 2024-09-18T01:54:18.478 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51692.log: /var/log/ceph/cluster2-client.mirror.61341.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51692.log.gz 2024-09-18T01:54:18.478 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54972.log 2024-09-18T01:54:18.478 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65234.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61341.log.gz 2024-09-18T01:54:18.478 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60076.log 2024-09-18T01:54:18.478 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64304.log 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60076.log: /var/log/ceph/cluster2-client.mirror.54972.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65234.log.gz 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71267.log 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60076.log.gz 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64304.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54972.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67705.log 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71267.log: /var/log/ceph/cluster2.audit.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71267.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47660.log 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67705.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67705.log.gz 2024-09-18T01:54:18.484 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47406.log 2024-09-18T01:54:18.485 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64304.log.gz 2024-09-18T01:54:18.485 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48530.log 2024-09-18T01:54:18.485 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64015.log 2024-09-18T01:54:18.485 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48530.log: /var/log/ceph/cluster2-client.mirror.47406.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48530.log.gz 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66670.log 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64015.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64015.log.gz 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66827.log 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66670.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66670.log.gz 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36966.log 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66827.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66827.log.gz 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44707.log 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.36966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37583.log 2024-09-18T01:54:18.489 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44707.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44707.log.gz 2024-09-18T01:54:18.492 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69021.log 2024-09-18T01:54:18.492 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51459.log 2024-09-18T01:54:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69021.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69021.log.gz 2024-09-18T01:54:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41180.log 2024-09-18T01:54:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51459.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51459.log.gz 2024-09-18T01:54:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63147.log 2024-09-18T01:54:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41180.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41180.log.gz 2024-09-18T01:54:18.494 INFO:teuthology.orchestra.run.smithi115.stderr: 92.4% 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37583.log.gz 2024-09-18T01:54:18.495 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36966.log.gz 2024-09-18T01:54:18.496 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72027.log 2024-09-18T01:54:18.497 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63147.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63147.log.gz 2024-09-18T01:54:18.498 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70816.log 2024-09-18T01:54:18.498 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.72027.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72027.log.gz 2024-09-18T01:54:18.499 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48672.log 2024-09-18T01:54:18.508 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57287.log 2024-09-18T01:54:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35953.log 2024-09-18T01:54:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57287.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48672.log.gz 2024-09-18T01:54:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49239.log 2024-09-18T01:54:18.529 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57287.log.gz 2024-09-18T01:54:18.529 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35953.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35953.log.gz 2024-09-18T01:54:18.535 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55662.log 2024-09-18T01:54:18.544 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39358.log 2024-09-18T01:54:18.548 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55662.log: 92.7% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-18T01:54:18.548 INFO:teuthology.orchestra.run.smithi115.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49239.log.gz 2024-09-18T01:54:18.548 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55662.log.gz 2024-09-18T01:54:18.558 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69245.log 2024-09-18T01:54:18.562 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39358.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45614.log.gz 2024-09-18T01:54:18.564 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36078.log 2024-09-18T01:54:18.564 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69245.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69245.log.gz 2024-09-18T01:54:18.564 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51439.log 2024-09-18T01:54:18.565 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36078.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36078.log.gz 2024-09-18T01:54:18.565 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44177.log 2024-09-18T01:54:18.565 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51439.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51439.log.gz 2024-09-18T01:54:18.565 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66131.log 2024-09-18T01:54:18.571 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44177.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44177.log.gz 2024-09-18T01:54:18.573 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39358.log.gz 2024-09-18T01:54:18.574 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65462.log 2024-09-18T01:54:18.574 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66131.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66131.log.gz 2024-09-18T01:54:18.574 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70207.log 2024-09-18T01:54:18.576 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65462.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65462.log.gz 2024-09-18T01:54:18.576 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71055.log 2024-09-18T01:54:18.576 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70207.log: 81.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70207.log.gz 2024-09-18T01:54:18.576 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43532.log 2024-09-18T01:54:18.576 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71055.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71055.log.gz 2024-09-18T01:54:18.584 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72360.log 2024-09-18T01:54:18.591 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43532.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41927.log.gz 2024-09-18T01:54:18.593 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63767.log 2024-09-18T01:54:18.593 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72360.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72360.log.gz 2024-09-18T01:54:18.593 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72988.log 2024-09-18T01:54:18.596 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63767.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63767.log.gz 2024-09-18T01:54:18.596 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36944.log 2024-09-18T01:54:18.596 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72988.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72988.log.gz 2024-09-18T01:54:18.596 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46327.log 2024-09-18T01:54:18.596 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.36944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64449.log 2024-09-18T01:54:18.599 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46327.log: 76.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36944.log.gz 2024-09-18T01:54:18.606 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.45742.log 2024-09-18T01:54:18.607 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64449.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64449.log.gz 2024-09-18T01:54:18.611 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43532.log.gz 2024-09-18T01:54:18.614 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.41544.log 2024-09-18T01:54:18.621 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.2.45742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43681.log 2024-09-18T01:54:18.631 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.1.41544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64945.log 2024-09-18T01:54:18.641 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43681.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66490.log 2024-09-18T01:54:18.642 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64945.log: -- replaced with /var/log/ceph/cluster1-client.mirror.1.41544.log.gz 2024-09-18T01:54:18.653 INFO:teuthology.orchestra.run.smithi115.stderr: 89.3% 93.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65338.log 2024-09-18T01:54:18.653 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66490.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66490.log.gz 2024-09-18T01:54:18.654 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64945.log.gz 2024-09-18T01:54:18.657 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.45742.log.gz 2024-09-18T01:54:18.660 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43681.log.gz 2024-09-18T01:54:18.661 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48291.log 2024-09-18T01:54:18.667 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65338.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65338.log.gz 2024-09-18T01:54:18.667 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62630.log 2024-09-18T01:54:18.668 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48291.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48291.log.gz 2024-09-18T01:54:18.668 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62411.log 2024-09-18T01:54:18.685 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56509.log 2024-09-18T01:54:18.686 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62411.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62411.log.gz 2024-09-18T01:54:18.694 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47266.log 2024-09-18T01:54:18.697 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56509.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69410.log.gz 2024-09-18T01:54:18.699 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62630.log.gz 2024-09-18T01:54:18.700 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56509.log.gz 2024-09-18T01:54:18.707 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57492.log 2024-09-18T01:54:18.707 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47266.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47266.log.gz 2024-09-18T01:54:18.707 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71353.log 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57492.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57492.log.gz 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53842.log 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71353.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71353.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54509.log 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60489.log 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.54509.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53842.log.gz 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60168.log 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60489.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60489.log.gz 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39843.log 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54509.log.gz 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46082.log 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39843.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60168.log.gz 2024-09-18T01:54:18.708 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49784.log 2024-09-18T01:54:18.709 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.46082.log: /var/log/ceph/cluster2-client.mirror.65400.log 2024-09-18T01:54:18.714 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49784.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49784.log.gz 2024-09-18T01:54:18.715 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52290.log 2024-09-18T01:54:18.716 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65400.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65400.log.gz 2024-09-18T01:54:18.716 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42874.log 2024-09-18T01:54:18.717 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52290.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52290.log.gz 2024-09-18T01:54:18.717 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66512.log 2024-09-18T01:54:18.723 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42874.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42874.log.gz 2024-09-18T01:54:18.724 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39863.log 2024-09-18T01:54:18.725 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66512.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66512.log.gz 2024-09-18T01:54:18.725 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66002.log 2024-09-18T01:54:18.727 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67300.log 2024-09-18T01:54:18.732 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.66002.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66002.log.gz 2024-09-18T01:54:18.733 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68421.log 2024-09-18T01:54:18.734 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67300.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67300.log.gz 2024-09-18T01:54:18.734 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48194.log 2024-09-18T01:54:18.737 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.68421.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68421.log.gz 2024-09-18T01:54:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48929.log 2024-09-18T01:54:18.739 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48194.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48194.log.gz 2024-09-18T01:54:18.739 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66962.log 2024-09-18T01:54:18.741 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48929.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48929.log.gz 2024-09-18T01:54:18.741 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62252.log 2024-09-18T01:54:18.742 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66962.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66962.log.gz 2024-09-18T01:54:18.742 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53730.log 2024-09-18T01:54:18.743 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62252.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62252.log.gz 2024-09-18T01:54:18.746 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39843.log.gz 2024-09-18T01:54:18.747 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67817.log 2024-09-18T01:54:18.750 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53730.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53730.log.gz 2024-09-18T01:54:18.750 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66400.log 2024-09-18T01:54:18.750 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67817.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67817.log.gz 2024-09-18T01:54:18.752 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.59543.log 2024-09-18T01:54:18.752 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66400.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66400.log.gz 2024-09-18T01:54:18.752 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44346.log 2024-09-18T01:54:18.761 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.1.59543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49032.log 2024-09-18T01:54:18.765 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44346.log: 89.7% 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44346.log.gz 2024-09-18T01:54:18.766 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.1.59543.log.gz 2024-09-18T01:54:18.766 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54468.log 2024-09-18T01:54:18.766 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49032.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49032.log.gz 2024-09-18T01:54:18.766 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66152.log 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54468.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54468.log.gz 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54278.log 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66152.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70816.log.gz 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66152.log.gz 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72240.log 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.42535.log 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.72240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72240.log.gz 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53244.log 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.0.42535.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54278.log.gz 2024-09-18T01:54:18.771 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64511.log 2024-09-18T01:54:18.773 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53244.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53244.log.gz 2024-09-18T01:54:18.773 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48513.log 2024-09-18T01:54:18.774 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64511.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64511.log.gz 2024-09-18T01:54:18.775 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51206.log 2024-09-18T01:54:18.776 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48513.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48513.log.gz 2024-09-18T01:54:18.776 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37378.log 2024-09-18T01:54:18.778 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51206.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51206.log.gz 2024-09-18T01:54:18.778 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70493.log 2024-09-18T01:54:18.778 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.37378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42811.log 2024-09-18T01:54:18.779 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70493.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70493.log.gz 2024-09-18T01:54:18.780 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46247.log 2024-09-18T01:54:18.780 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42811.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37378.log.gz 2024-09-18T01:54:18.786 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42811.log.gz 2024-09-18T01:54:18.786 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36249.log 2024-09-18T01:54:18.804 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49965.log 2024-09-18T01:54:18.804 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47660.log.gz 2024-09-18T01:54:18.804 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36249.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60947.log 2024-09-18T01:54:18.804 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.36249.log.gz 2024-09-18T01:54:18.804 INFO:teuthology.orchestra.run.smithi115.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.42535.log.gz 2024-09-18T01:54:18.804 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46811.log 2024-09-18T01:54:18.804 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60947.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49965.log.gz 2024-09-18T01:54:18.804 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49256.log 2024-09-18T01:54:18.812 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46811.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60947.log.gz 2024-09-18T01:54:18.812 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44091.log 2024-09-18T01:54:18.817 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49256.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49256.log.gz 2024-09-18T01:54:18.817 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43767.log 2024-09-18T01:54:18.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45143.log 2024-09-18T01:54:18.837 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43767.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44091.log.gz 2024-09-18T01:54:18.838 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43767.log.gz 2024-09-18T01:54:18.838 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66805.log 2024-09-18T01:54:18.838 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45143.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45143.log.gz 2024-09-18T01:54:18.838 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60329.log 2024-09-18T01:54:18.839 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66805.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66805.log.gz 2024-09-18T01:54:18.839 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51895.log 2024-09-18T01:54:18.841 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60329.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60329.log.gz 2024-09-18T01:54:18.841 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53962.log 2024-09-18T01:54:18.843 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51895.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51895.log.gz 2024-09-18T01:54:18.843 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45592.log 2024-09-18T01:54:18.844 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53962.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53962.log.gz 2024-09-18T01:54:18.844 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64883.log 2024-09-18T01:54:18.846 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45592.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45592.log.gz 2024-09-18T01:54:18.846 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40593.log 2024-09-18T01:54:18.847 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64883.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64883.log.gz 2024-09-18T01:54:18.847 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64635.log 2024-09-18T01:54:18.848 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.40593.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40593.log.gz 2024-09-18T01:54:18.849 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45524.log 2024-09-18T01:54:18.850 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64635.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64635.log.gz 2024-09-18T01:54:18.850 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53131.log 2024-09-18T01:54:18.867 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.36789.log 2024-09-18T01:54:18.868 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53131.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53131.log.gz 2024-09-18T01:54:18.881 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40212.log 2024-09-18T01:54:18.888 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.2.36789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47681.log 2024-09-18T01:54:18.895 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47718.log 2024-09-18T01:54:18.897 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47681.log: 80.4% 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40212.log.gz 2024-09-18T01:54:18.902 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45974.log 2024-09-18T01:54:18.909 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47718.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47681.log.gz 2024-09-18T01:54:18.910 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57974.log 2024-09-18T01:54:18.916 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45974.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47718.log.gz 2024-09-18T01:54:18.922 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48333.log 2024-09-18T01:54:18.922 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45974.log.gz 2024-09-18T01:54:18.923 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57974.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57974.log.gz 2024-09-18T01:54:18.929 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59843.log 2024-09-18T01:54:18.934 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48333.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47406.log.gz 2024-09-18T01:54:18.936 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59959.log 2024-09-18T01:54:18.941 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59843.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59843.log.gz 2024-09-18T01:54:18.943 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63912.log 2024-09-18T01:54:18.943 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59959.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59959.log.gz 2024-09-18T01:54:18.943 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68155.log 2024-09-18T01:54:18.944 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63912.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63912.log.gz 2024-09-18T01:54:18.946 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43466.log 2024-09-18T01:54:18.946 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68155.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68155.log.gz 2024-09-18T01:54:18.947 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39863.log.gz 2024-09-18T01:54:18.950 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46327.log.gz 2024-09-18T01:54:18.951 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46123.log 2024-09-18T01:54:18.957 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43466.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43466.log.gz 2024-09-18T01:54:18.958 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48731.log 2024-09-18T01:54:18.960 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46123.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46123.log.gz 2024-09-18T01:54:18.960 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70674.log 2024-09-18T01:54:18.974 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55546.log 2024-09-18T01:54:18.974 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70674.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70674.log.gz 2024-09-18T01:54:18.986 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48652.log.gz 2024-09-18T01:54:18.987 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37429.log 2024-09-18T01:54:18.988 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55546.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55546.log.gz 2024-09-18T01:54:18.990 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.40476.log 2024-09-18T01:54:18.990 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.37429.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37429.log.gz 2024-09-18T01:54:18.990 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71937.log 2024-09-18T01:54:18.996 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.0.40476.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.40476.log.gz 2024-09-18T01:54:18.996 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71394.log 2024-09-18T01:54:18.997 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71937.log: 86.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71937.log.gz 2024-09-18T01:54:18.997 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47638.log 2024-09-18T01:54:19.000 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71394.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71394.log.gz 2024-09-18T01:54:19.001 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48114.log 2024-09-18T01:54:19.002 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47638.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47638.log.gz 2024-09-18T01:54:19.002 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63044.log 2024-09-18T01:54:19.003 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48114.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48114.log.gz 2024-09-18T01:54:19.003 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64408.log 2024-09-18T01:54:19.004 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63044.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63044.log.gz 2024-09-18T01:54:19.004 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52314.log 2024-09-18T01:54:19.006 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64408.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64408.log.gz 2024-09-18T01:54:19.006 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59662.log 2024-09-18T01:54:19.007 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52314.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52314.log.gz 2024-09-18T01:54:19.007 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40911.log 2024-09-18T01:54:19.009 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59662.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59662.log.gz 2024-09-18T01:54:19.009 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55084.log 2024-09-18T01:54:19.011 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40911.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40911.log.gz 2024-09-18T01:54:19.011 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65857.log 2024-09-18T01:54:19.012 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55084.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55084.log.gz 2024-09-18T01:54:19.012 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55915.log 2024-09-18T01:54:19.014 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65857.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65857.log.gz 2024-09-18T01:54:19.014 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58344.log 2024-09-18T01:54:19.015 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55915.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55915.log.gz 2024-09-18T01:54:19.015 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71518.log 2024-09-18T01:54:19.017 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58344.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58344.log.gz 2024-09-18T01:54:19.017 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47159.log 2024-09-18T01:54:19.019 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71518.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71518.log.gz 2024-09-18T01:54:19.019 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.68887.log 2024-09-18T01:54:19.026 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47159.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46082.log.gz 2024-09-18T01:54:19.027 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49618.log 2024-09-18T01:54:19.033 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.3.68887.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.68887.log.gz 2024-09-18T01:54:19.033 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42694.log 2024-09-18T01:54:19.034 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49618.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49618.log.gz 2024-09-18T01:54:19.035 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65919.log 2024-09-18T01:54:19.035 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42694.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42694.log.gz 2024-09-18T01:54:19.036 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43747.log 2024-09-18T01:54:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65919.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65919.log.gz 2024-09-18T01:54:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72862.log 2024-09-18T01:54:19.044 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43747.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43747.log.gz 2024-09-18T01:54:19.045 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35787.log 2024-09-18T01:54:19.045 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72862.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72862.log.gz 2024-09-18T01:54:19.047 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36333.log 2024-09-18T01:54:19.047 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35787.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35787.log.gz 2024-09-18T01:54:19.048 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59011.log 2024-09-18T01:54:19.048 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.36333.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36333.log.gz 2024-09-18T01:54:19.048 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51326.log 2024-09-18T01:54:19.049 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59011.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59011.log.gz 2024-09-18T01:54:19.049 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44797.log 2024-09-18T01:54:19.049 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51326.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51326.log.gz 2024-09-18T01:54:19.051 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70774.log 2024-09-18T01:54:19.051 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44797.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44797.log.gz 2024-09-18T01:54:19.051 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62493.log 2024-09-18T01:54:19.053 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70774.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70774.log.gz 2024-09-18T01:54:19.053 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65379.log 2024-09-18T01:54:19.054 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62493.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62493.log.gz 2024-09-18T01:54:19.055 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41364.log 2024-09-18T01:54:19.055 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65379.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65379.log.gz 2024-09-18T01:54:19.056 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57697.log 2024-09-18T01:54:19.063 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41364.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41364.log.gz 2024-09-18T01:54:19.063 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37617.log 2024-09-18T01:54:19.064 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57697.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57697.log.gz 2024-09-18T01:54:19.064 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43332.log 2024-09-18T01:54:19.065 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67277.log 2024-09-18T01:54:19.066 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43332.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43332.log.gz 2024-09-18T01:54:19.067 INFO:teuthology.orchestra.run.smithi115.stderr: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37617.log.gz 2024-09-18T01:54:19.068 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67997.log 2024-09-18T01:54:19.069 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67277.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67277.log.gz 2024-09-18T01:54:19.069 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60582.log 2024-09-18T01:54:19.070 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67997.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67997.log.gz 2024-09-18T01:54:19.070 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.45885.log 2024-09-18T01:54:19.072 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60582.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60582.log.gz 2024-09-18T01:54:19.073 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56880.log 2024-09-18T01:54:19.079 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.3.45885.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.45885.log.gz 2024-09-18T01:54:19.080 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71288.log 2024-09-18T01:54:19.080 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56880.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56880.log.gz 2024-09-18T01:54:19.081 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55064.log 2024-09-18T01:54:19.082 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71288.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71288.log.gz 2024-09-18T01:54:19.083 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69499.log 2024-09-18T01:54:19.084 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55064.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55064.log.gz 2024-09-18T01:54:19.084 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62960.log 2024-09-18T01:54:19.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69499.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69499.log.gz 2024-09-18T01:54:19.085 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46789.log 2024-09-18T01:54:19.086 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62960.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62960.log.gz 2024-09-18T01:54:19.087 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66332.log 2024-09-18T01:54:19.087 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46789.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46789.log.gz 2024-09-18T01:54:19.088 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64222.log 2024-09-18T01:54:19.089 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66332.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66332.log.gz 2024-09-18T01:54:19.089 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53822.log 2024-09-18T01:54:19.089 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64222.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64222.log.gz 2024-09-18T01:54:19.090 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44586.log 2024-09-18T01:54:19.091 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53822.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53822.log.gz 2024-09-18T01:54:19.092 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58528.log 2024-09-18T01:54:19.092 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44586.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44586.log.gz 2024-09-18T01:54:19.093 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42854.log 2024-09-18T01:54:19.093 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58528.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58528.log.gz 2024-09-18T01:54:19.094 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41703.log 2024-09-18T01:54:19.100 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42854.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42854.log.gz 2024-09-18T01:54:19.100 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45636.log 2024-09-18T01:54:19.101 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41703.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41703.log.gz 2024-09-18T01:54:19.101 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53772.log 2024-09-18T01:54:19.102 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45636.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45636.log.gz 2024-09-18T01:54:19.102 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60212.log 2024-09-18T01:54:19.103 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53772.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53772.log.gz 2024-09-18T01:54:19.104 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46464.log 2024-09-18T01:54:19.104 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60212.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60212.log.gz 2024-09-18T01:54:19.105 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39040.log 2024-09-18T01:54:19.105 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46464.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46464.log.gz 2024-09-18T01:54:19.106 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58299.log 2024-09-18T01:54:19.106 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68357.log 2024-09-18T01:54:19.107 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58299.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58299.log.gz 2024-09-18T01:54:19.114 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67570.log 2024-09-18T01:54:19.115 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68357.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68357.log.gz 2024-09-18T01:54:19.117 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64862.log 2024-09-18T01:54:19.117 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67570.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67570.log.gz 2024-09-18T01:54:19.118 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60236.log 2024-09-18T01:54:19.120 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70165.log.gz 2024-09-18T01:54:19.123 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64862.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64862.log.gz 2024-09-18T01:54:19.125 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60236.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60236.log.gz 2024-09-18T01:54:19.125 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40889.log 2024-09-18T01:54:19.125 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59682.log 2024-09-18T01:54:19.128 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58689.log 2024-09-18T01:54:19.128 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59682.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59682.log.gz 2024-09-18T01:54:19.129 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56422.log 2024-09-18T01:54:19.130 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58689.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58689.log.gz 2024-09-18T01:54:19.131 INFO:teuthology.orchestra.run.smithi115.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40889.log.gz 2024-09-18T01:54:19.132 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43489.log 2024-09-18T01:54:19.132 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56422.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56422.log.gz 2024-09-18T01:54:19.133 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46143.log 2024-09-18T01:54:19.133 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71731.log 2024-09-18T01:54:19.134 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43489.log: /var/log/ceph/cluster2-client.mirror.46143.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46143.log.gz 2024-09-18T01:54:19.135 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66197.log 2024-09-18T01:54:19.135 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71731.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71731.log.gz 2024-09-18T01:54:19.138 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63106.log 2024-09-18T01:54:19.138 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66197.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66197.log.gz 2024-09-18T01:54:19.139 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43489.log.gz 2024-09-18T01:54:19.140 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51555.log 2024-09-18T01:54:19.140 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63106.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63106.log.gz 2024-09-18T01:54:19.142 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53661.log 2024-09-18T01:54:19.142 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70935.log 2024-09-18T01:54:19.142 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51555.log.gz 2024-09-18T01:54:19.144 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60260.log 2024-09-18T01:54:19.144 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70935.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53661.log.gz 2024-09-18T01:54:19.144 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70935.log.gz 2024-09-18T01:54:19.144 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63416.log/var/log/ceph/cluster1-client.mirror.60260.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60260.log.gz 2024-09-18T01:54:19.146 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:19.147 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65048.log 2024-09-18T01:54:19.147 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63416.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63416.log.gz 2024-09-18T01:54:19.150 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65048.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65048.log.gz 2024-09-18T01:54:19.151 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47222.log 2024-09-18T01:54:19.153 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56120.log 2024-09-18T01:54:19.153 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58576.log 2024-09-18T01:54:19.153 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47222.log.gz 2024-09-18T01:54:19.153 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44446.log 2024-09-18T01:54:19.153 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58576.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56120.log.gz 2024-09-18T01:54:19.153 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58576.log.gz 2024-09-18T01:54:19.155 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44446.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44446.log.gz 2024-09-18T01:54:19.155 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42742.log 2024-09-18T01:54:19.156 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42422.log 2024-09-18T01:54:19.156 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42742.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42742.log.gz 2024-09-18T01:54:19.156 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38743.log 2024-09-18T01:54:19.157 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53408.log 2024-09-18T01:54:19.159 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.38743.log: /var/log/ceph/cluster1-client.mirror.53408.log: /var/log/ceph/cluster2-client.mirror.42422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46871.log 2024-09-18T01:54:19.162 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53408.log.gz 2024-09-18T01:54:19.162 INFO:teuthology.orchestra.run.smithi115.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38743.log.gz 2024-09-18T01:54:19.162 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69670.log 2024-09-18T01:54:19.162 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46871.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46871.log.gz 2024-09-18T01:54:19.162 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64986.log 2024-09-18T01:54:19.162 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69670.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69670.log.gz 2024-09-18T01:54:19.163 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42154.log 2024-09-18T01:54:19.165 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64986.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64986.log.gz 2024-09-18T01:54:19.167 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42154.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42154.log.gz 2024-09-18T01:54:19.169 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45403.log 2024-09-18T01:54:19.169 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-18T01:54:19.169 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45403.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45403.log.gz 2024-09-18T01:54:19.191 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54763.log 2024-09-18T01:54:19.200 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42270.log 2024-09-18T01:54:19.207 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66872.log 2024-09-18T01:54:19.209 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42270.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54763.log.gz 2024-09-18T01:54:19.215 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44727.log 2024-09-18T01:54:19.215 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66872.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42270.log.gz 2024-09-18T01:54:19.216 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66872.log.gz 2024-09-18T01:54:19.227 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55385.log 2024-09-18T01:54:19.236 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70695.log 2024-09-18T01:54:19.237 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55385.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55385.log.gz 2024-09-18T01:54:19.243 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66022.log 2024-09-18T01:54:19.243 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70695.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70695.log.gz 2024-09-18T01:54:19.253 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66895.log 2024-09-18T01:54:19.262 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.66022.log: 90.3% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39486.log.gz 2024-09-18T01:54:19.266 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66022.log.gz 2024-09-18T01:54:19.267 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48731.log.gz 2024-09-18T01:54:19.271 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55843.log 2024-09-18T01:54:19.272 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66895.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66895.log.gz 2024-09-18T01:54:19.272 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38835.log 2024-09-18T01:54:19.273 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55843.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55843.log.gz 2024-09-18T01:54:19.274 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45160.log 2024-09-18T01:54:19.275 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.38835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53063.log 2024-09-18T01:54:19.276 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45160.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45160.log.gz 2024-09-18T01:54:19.277 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66917.log 2024-09-18T01:54:19.277 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53063.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53063.log.gz 2024-09-18T01:54:19.280 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63705.log 2024-09-18T01:54:19.284 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66917.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66917.log.gz 2024-09-18T01:54:19.284 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60441.log 2024-09-18T01:54:19.286 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63705.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63705.log.gz 2024-09-18T01:54:19.286 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65524.log 2024-09-18T01:54:19.286 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60441.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60441.log.gz 2024-09-18T01:54:19.290 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48572.log 2024-09-18T01:54:19.290 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65524.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65524.log.gz 2024-09-18T01:54:19.290 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.48025.log 2024-09-18T01:54:19.290 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48572.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38835.log.gz 2024-09-18T01:54:19.293 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48333.log.gz 2024-09-18T01:54:19.293 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52495.log 2024-09-18T01:54:19.297 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.0.48025.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.48025.log.gz 2024-09-18T01:54:19.297 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41883.log 2024-09-18T01:54:19.298 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52495.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52495.log.gz 2024-09-18T01:54:19.298 INFO:teuthology.orchestra.run.smithi115.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.36789.log.gz 2024-09-18T01:54:19.298 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40566.log 2024-09-18T01:54:19.299 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56072.log 2024-09-18T01:54:19.300 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.40566.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40566.log.gz 2024-09-18T01:54:19.300 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52741.log 2024-09-18T01:54:19.301 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56072.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56072.log.gz 2024-09-18T01:54:19.302 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54236.log 2024-09-18T01:54:19.303 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52741.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52741.log.gz 2024-09-18T01:54:19.303 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39963.log 2024-09-18T01:54:19.307 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54236.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54236.log.gz 2024-09-18T01:54:19.307 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59035.log 2024-09-18T01:54:19.307 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.39963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46544.log 2024-09-18T01:54:19.308 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4%/var/log/ceph/cluster1-client.mirror.59035.log: -- replaced with /var/log/ceph/cluster1-client.mirror.41883.log.gz 2024-09-18T01:54:19.308 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55293.log 2024-09-18T01:54:19.308 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59035.log.gz 2024-09-18T01:54:19.308 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46544.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.39963.log.gz 2024-09-18T01:54:19.308 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46544.log.gz 2024-09-18T01:54:19.308 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.41964.log 2024-09-18T01:54:19.310 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55293.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55293.log.gz 2024-09-18T01:54:19.310 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65482.log 2024-09-18T01:54:19.310 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.1.41964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69180.log 2024-09-18T01:54:19.319 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65482.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65482.log.gz 2024-09-18T01:54:19.319 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60308.log 2024-09-18T01:54:19.322 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69180.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69180.log.gz 2024-09-18T01:54:19.325 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36921.log 2024-09-18T01:54:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60308.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60308.log.gz 2024-09-18T01:54:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43638.log 2024-09-18T01:54:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.36921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47783.log 2024-09-18T01:54:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43638.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.41964.log.gz 2024-09-18T01:54:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58275.log 2024-09-18T01:54:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47783.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47783.log.gz 2024-09-18T01:54:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61088.log 2024-09-18T01:54:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58275.log: 74.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36921.log.gz 2024-09-18T01:54:19.333 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58275.log.gz 2024-09-18T01:54:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44727.log.gz 2024-09-18T01:54:19.333 INFO:teuthology.orchestra.run.smithi115.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.51278.log 2024-09-18T01:54:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61088.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61088.log.gz 2024-09-18T01:54:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68267.log 2024-09-18T01:54:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51278.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51278.log.gz 2024-09-18T01:54:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:gzip 93.4% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48832.log 2024-09-18T01:54:19.339 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43638.log.gz 2024-09-18T01:54:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57311.log 2024-09-18T01:54:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68267.log: /var/log/ceph/cluster2-client.mirror.48832.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68267.log.gz 2024-09-18T01:54:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43575.log 2024-09-18T01:54:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57311.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57311.log.gz 2024-09-18T01:54:19.340 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62897.log 2024-09-18T01:54:19.341 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43575.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48832.log.gz 2024-09-18T01:54:19.341 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66602.log 2024-09-18T01:54:19.342 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46247.log.gz 2024-09-18T01:54:19.344 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62897.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62897.log.gz 2024-09-18T01:54:19.355 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43575.log.gz 2024-09-18T01:54:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55819.log 2024-09-18T01:54:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69580.log 2024-09-18T01:54:19.356 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66602.log.gz 2024-09-18T01:54:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48869.log 2024-09-18T01:54:19.356 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55819.log.gz/var/log/ceph/cluster1-client.mirror.69580.log: 2024-09-18T01:54:19.356 INFO:teuthology.orchestra.run.smithi115.stderr: 85.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69580.log.gz 2024-09-18T01:54:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43939.log 2024-09-18T01:54:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48869.log: /var/log/ceph/cluster1-client.mirror.43939.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48869.log.gz 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46769.log 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44486.log 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.46769.log: -- /var/log/ceph/cluster1-client.mirror.37395.log 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44486.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46769.log.gz 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42766.log 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44486.log.gz/var/log/ceph/cluster1-client.mirror.37395.log: 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62918.log 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41160.log 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62918.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62918.log.gz 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54420.log 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43939.log.gz 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41160.log: /var/log/ceph/cluster1-client.mirror.42766.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42766.log.gz 2024-09-18T01:54:19.357 INFO:teuthology.orchestra.run.smithi115.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41160.log.gz 2024-09-18T01:54:19.358 INFO:teuthology.orchestra.run.smithi115.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37395.log.gz 2024-09-18T01:54:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48253.log 2024-09-18T01:54:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44566.log 2024-09-18T01:54:19.358 INFO:teuthology.orchestra.run.smithi115.stderr: 87.2%/var/log/ceph/cluster2-client.mirror.48253.log: -- replaced with /var/log/ceph/cluster1-client.mirror.54420.log.gz 2024-09-18T01:54:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56052.log 2024-09-18T01:54:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68177.log 2024-09-18T01:54:19.358 INFO:teuthology.orchestra.run.smithi115.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44566.log.gz 2024-09-18T01:54:19.361 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68177.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68177.log.gz 2024-09-18T01:54:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54167.log 2024-09-18T01:54:19.362 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56052.log.gz 2024-09-18T01:54:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43618.log 2024-09-18T01:54:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54003.log 2024-09-18T01:54:19.362 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54167.log.gz 2024-09-18T01:54:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63891.log 2024-09-18T01:54:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54003.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54003.log.gz 2024-09-18T01:54:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63891.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63891.log.gz 2024-09-18T01:54:19.370 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45240.log 2024-09-18T01:54:19.370 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39567.log 2024-09-18T01:54:19.370 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45240.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43618.log.gz 2024-09-18T01:54:19.370 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70891.log 2024-09-18T01:54:19.370 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.39567.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55939.log 2024-09-18T01:54:19.370 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45240.log.gz 2024-09-18T01:54:19.370 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52950.log 2024-09-18T01:54:19.371 INFO:teuthology.orchestra.run.smithi115.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70891.log.gz 2024-09-18T01:54:19.372 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55939.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55939.log.gz 2024-09-18T01:54:19.374 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52950.log: gzip 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52950.log.gz 2024-09-18T01:54:19.374 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56603.log 2024-09-18T01:54:19.375 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70414.log 2024-09-18T01:54:19.375 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56603.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56603.log.gz 2024-09-18T01:54:19.376 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50861.log 2024-09-18T01:54:19.377 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70414.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70414.log.gz 2024-09-18T01:54:19.377 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55180.log 2024-09-18T01:54:19.378 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71495.log 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55180.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50861.log.gz 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55180.log.gz 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.40053.log 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39567.log.gz 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58597.log 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.2.40053.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71495.log.gz 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61200.log 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65712.log 2024-09-18T01:54:19.383 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61200.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61200.log.gz 2024-09-18T01:54:19.385 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46406.log/var/log/ceph/cluster2-client.mirror.65712.log: 2024-09-18T01:54:19.385 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58597.log.gz 2024-09-18T01:54:19.389 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45524.log.gz 2024-09-18T01:54:19.391 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67547.log 2024-09-18T01:54:19.397 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65712.log.gz 2024-09-18T01:54:19.399 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48593.log 2024-09-18T01:54:19.401 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67547.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67547.log.gz 2024-09-18T01:54:19.401 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56398.log 2024-09-18T01:54:19.405 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48593.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48593.log.gz 2024-09-18T01:54:19.405 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66940.log 2024-09-18T01:54:19.405 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56398.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56398.log.gz 2024-09-18T01:54:19.415 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41341.log 2024-09-18T01:54:19.415 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66940.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66940.log.gz 2024-09-18T01:54:19.415 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51784.log 2024-09-18T01:54:19.424 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62652.log 2024-09-18T01:54:19.433 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51784.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40733.log 2024-09-18T01:54:19.433 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51784.log.gz 2024-09-18T01:54:19.434 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62652.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41341.log.gz 2024-09-18T01:54:19.434 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51736.log 2024-09-18T01:54:19.438 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40733.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40733.log.gz 2024-09-18T01:54:19.442 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51808.log 2024-09-18T01:54:19.443 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51736.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51736.log.gz 2024-09-18T01:54:19.444 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61525.log 2024-09-18T01:54:19.444 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62652.log.gz 2024-09-18T01:54:19.444 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51808.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51808.log.gz 2024-09-18T01:54:19.448 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63746.log 2024-09-18T01:54:19.452 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61525.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61525.log.gz 2024-09-18T01:54:19.452 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49054.log 2024-09-18T01:54:19.464 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69543.log 2024-09-18T01:54:19.476 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65441.log 2024-09-18T01:54:19.478 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69543.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63746.log.gz 2024-09-18T01:54:19.478 INFO:teuthology.orchestra.run.smithi115.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69543.log.gz 2024-09-18T01:54:19.489 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42994.log 2024-09-18T01:54:19.490 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65441.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65441.log.gz 2024-09-18T01:54:19.503 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68200.log 2024-09-18T01:54:19.504 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42994.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42994.log.gz 2024-09-18T01:54:19.518 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42422.log.gz 2024-09-18T01:54:19.518 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71432.log 2024-09-18T01:54:19.518 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68200.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68200.log.gz 2024-09-18T01:54:19.520 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65940.log 2024-09-18T01:54:19.535 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71432.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71432.log.gz 2024-09-18T01:54:19.539 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.36749.log 2024-09-18T01:54:19.540 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65940.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65940.log.gz 2024-09-18T01:54:19.543 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69996.log 2024-09-18T01:54:19.544 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.1.36749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46914.log 2024-09-18T01:54:19.544 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69996.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69996.log.gz 2024-09-18T01:54:19.545 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57109.log 2024-09-18T01:54:19.552 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46914.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48572.log.gz 2024-09-18T01:54:19.552 INFO:teuthology.orchestra.run.smithi115.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46914.log.gz 2024-09-18T01:54:19.554 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44646.log 2024-09-18T01:54:19.554 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57109.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57109.log.gz 2024-09-18T01:54:19.554 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66084.log 2024-09-18T01:54:19.555 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44646.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44646.log.gz 2024-09-18T01:54:19.555 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60188.log 2024-09-18T01:54:19.556 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.66084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57721.log 2024-09-18T01:54:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60188.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60188.log.gz 2024-09-18T01:54:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70393.log 2024-09-18T01:54:19.562 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57721.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57721.log.gz 2024-09-18T01:54:19.562 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47847.log 2024-09-18T01:54:19.562 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66084.log.gz 2024-09-18T01:54:19.562 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44283.log 2024-09-18T01:54:19.562 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47847.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70393.log.gz 2024-09-18T01:54:19.563 INFO:teuthology.orchestra.run.smithi115.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47847.log.gz 2024-09-18T01:54:19.563 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52061.log 2024-09-18T01:54:19.563 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.44283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47618.log 2024-09-18T01:54:19.563 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52061.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52061.log.gz 2024-09-18T01:54:19.563 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61320.log 2024-09-18T01:54:19.565 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47618.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47618.log.gz 2024-09-18T01:54:19.565 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60096.log 2024-09-18T01:54:19.565 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61320.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61320.log.gz 2024-09-18T01:54:19.565 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69120.log 2024-09-18T01:54:19.565 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60096.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44283.log.gz 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59865.log 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69120.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69120.log.gz 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52379.log 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59865.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60096.log.gz 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr:gzip 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59865.log.gz 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55337.log 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56168.log 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55337.log: /var/log/ceph/cluster1-client.mirror.52379.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55337.log.gz 2024-09-18T01:54:19.570 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39310.log 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56168.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52379.log.gz 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48151.log 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56762.log 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48151.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48151.log.gz 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56168.log.gz 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49380.log 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47966.log 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56762.log: /var/log/ceph/cluster2-client.mirror.49380.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56762.log.gz 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58022.log 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47966.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47966.log.gz 2024-09-18T01:54:19.580 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64118.log 2024-09-18T01:54:19.581 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58022.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58022.log.gz 2024-09-18T01:54:19.581 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.52741.log 2024-09-18T01:54:19.581 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39310.log.gz 2024-09-18T01:54:19.582 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64118.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64118.log.gz 2024-09-18T01:54:19.582 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48231.log 2024-09-18T01:54:19.592 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.3.52741.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.52741.log.gz 2024-09-18T01:54:19.592 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67120.log 2024-09-18T01:54:19.592 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48231.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48231.log.gz 2024-09-18T01:54:19.592 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62541.log 2024-09-18T01:54:19.592 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67120.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67120.log.gz 2024-09-18T01:54:19.592 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35827.log 2024-09-18T01:54:19.595 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62541.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62541.log.gz 2024-09-18T01:54:19.595 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69813.log 2024-09-18T01:54:19.595 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35827.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35827.log.gz 2024-09-18T01:54:19.596 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.65602.log 2024-09-18T01:54:19.596 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69813.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69813.log.gz 2024-09-18T01:54:19.596 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71900.log 2024-09-18T01:54:19.604 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.1.65602.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.65602.log.gz 2024-09-18T01:54:19.604 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69714.log 2024-09-18T01:54:19.612 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71900.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71900.log.gz 2024-09-18T01:54:19.612 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42718.log 2024-09-18T01:54:19.612 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69714.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69714.log.gz 2024-09-18T01:54:19.612 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60537.log 2024-09-18T01:54:19.612 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42718.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42718.log.gz 2024-09-18T01:54:19.612 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67165.log 2024-09-18T01:54:19.613 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60537.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60537.log.gz 2024-09-18T01:54:19.613 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44197.log 2024-09-18T01:54:19.613 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67165.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67165.log.gz 2024-09-18T01:54:19.613 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41228.log 2024-09-18T01:54:19.615 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.44197.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.36749.log.gz 2024-09-18T01:54:19.618 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44197.log.gz 2024-09-18T01:54:19.618 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59302.log 2024-09-18T01:54:19.618 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41228.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41228.log.gz 2024-09-18T01:54:19.619 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52037.log 2024-09-18T01:54:19.620 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59302.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59302.log.gz 2024-09-18T01:54:19.621 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64532.log 2024-09-18T01:54:19.622 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52037.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52037.log.gz 2024-09-18T01:54:19.622 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35308.log 2024-09-18T01:54:19.624 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64532.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64532.log.gz 2024-09-18T01:54:19.625 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46427.log 2024-09-18T01:54:19.625 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35308.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35308.log.gz 2024-09-18T01:54:19.625 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61296.log 2024-09-18T01:54:19.625 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46427.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46427.log.gz 2024-09-18T01:54:19.627 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71539.log 2024-09-18T01:54:19.627 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61296.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61296.log.gz 2024-09-18T01:54:19.627 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61865.log 2024-09-18T01:54:19.628 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71539.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71539.log.gz 2024-09-18T01:54:19.629 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50883.log 2024-09-18T01:54:19.629 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61865.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61865.log.gz 2024-09-18T01:54:19.630 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43810.log 2024-09-18T01:54:19.631 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50883.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50883.log.gz 2024-09-18T01:54:19.632 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50335.log 2024-09-18T01:54:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43810.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43810.log.gz 2024-09-18T01:54:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45848.log 2024-09-18T01:54:19.641 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50335.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50335.log.gz 2024-09-18T01:54:19.641 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66355.log 2024-09-18T01:54:19.642 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45848.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45848.log.gz 2024-09-18T01:54:19.642 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43040.log 2024-09-18T01:54:19.644 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66355.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66355.log.gz 2024-09-18T01:54:19.644 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35577.log 2024-09-18T01:54:19.646 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43040.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43040.log.gz 2024-09-18T01:54:19.646 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66985.log 2024-09-18T01:54:19.646 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35577.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35577.log.gz 2024-09-18T01:54:19.646 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36627.log 2024-09-18T01:54:19.648 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66985.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66985.log.gz 2024-09-18T01:54:19.649 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49358.log 2024-09-18T01:54:19.649 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36627.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36627.log.gz 2024-09-18T01:54:19.649 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54626.log 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49358.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49358.log.gz 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54903.log 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46811.log.gz 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54626.log: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43424.log 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54626.log.gz/var/log/ceph/cluster1-client.mirror.54903.log: 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr:gzip 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54903.log.gz 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49921.log 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50447.log 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49921.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43424.log.gz 2024-09-18T01:54:19.655 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44154.log 2024-09-18T01:54:19.656 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50447.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49921.log.gz 2024-09-18T01:54:19.656 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50447.log.gz 2024-09-18T01:54:19.656 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62449.log 2024-09-18T01:54:19.656 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.44154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43790.log 2024-09-18T01:54:19.656 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62449.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62449.log.gz 2024-09-18T01:54:19.657 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56487.log 2024-09-18T01:54:19.659 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43790.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44154.log.gz 2024-09-18T01:54:19.660 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57174.log 2024-09-18T01:54:19.661 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56487.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43790.log.gz 2024-09-18T01:54:19.662 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54722.log 2024-09-18T01:54:19.662 INFO:teuthology.orchestra.run.smithi115.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56487.log.gz 2024-09-18T01:54:19.664 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61043.log 2024-09-18T01:54:19.664 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54722.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54722.log.gz 2024-09-18T01:54:19.664 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56627.log 2024-09-18T01:54:19.666 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61043.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61043.log.gz 2024-09-18T01:54:19.666 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51965.log 2024-09-18T01:54:19.666 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56627.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56627.log.gz 2024-09-18T01:54:19.670 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-18T01:54:19.670 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51965.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51965.log.gz 2024-09-18T01:54:19.670 INFO:teuthology.orchestra.run.smithi115.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.40053.log.gz 2024-09-18T01:54:19.670 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53569.log 2024-09-18T01:54:19.670 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49941.log 2024-09-18T01:54:19.671 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44048.log 2024-09-18T01:54:19.671 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49941.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49941.log.gz 2024-09-18T01:54:19.671 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53569.log.gz 2024-09-18T01:54:19.671 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61549.log 2024-09-18T01:54:19.675 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44048.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44048.log.gz 2024-09-18T01:54:19.676 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67772.log 2024-09-18T01:54:19.677 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61549.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61549.log.gz 2024-09-18T01:54:19.678 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49338.log 2024-09-18T01:54:19.679 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67772.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67772.log.gz 2024-09-18T01:54:19.679 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62565.log 2024-09-18T01:54:19.682 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49338.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49338.log.gz 2024-09-18T01:54:19.682 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64821.log 2024-09-18T01:54:19.682 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62565.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62565.log.gz 2024-09-18T01:54:19.682 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47906.log 2024-09-18T01:54:19.683 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64821.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64821.log.gz 2024-09-18T01:54:19.683 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71018.log 2024-09-18T01:54:19.702 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47906.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57174.log.gz 2024-09-18T01:54:19.705 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37412.log 2024-09-18T01:54:19.706 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71018.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71018.log.gz 2024-09-18T01:54:19.706 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53589.log 2024-09-18T01:54:19.706 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.37412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63788.log 2024-09-18T01:54:19.706 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37327.log 2024-09-18T01:54:19.706 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53589.log.gz 2024-09-18T01:54:19.706 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63788.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63788.log.gz 2024-09-18T01:54:19.706 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48274.log 2024-09-18T01:54:19.711 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.37327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40982.log 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48274.log: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37412.log.gz 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48274.log.gz 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47185.log 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37327.log.gz 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.40982.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40982.log.gz 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68977.log 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47185.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47185.log.gz 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66043.log 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68977.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68977.log.gz 2024-09-18T01:54:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57858.log 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.66043.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46406.log.gz 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66043.log.gz 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48391.log 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57858.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55014.log 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53203.log 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48391.log.gz 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55014.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57858.log.gz 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55014.log.gz 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50796.log 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39731.log 2024-09-18T01:54:19.719 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53203.log.gz/var/log/ceph/cluster1-client.mirror.50796.log: 2024-09-18T01:54:19.720 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40260.log 2024-09-18T01:54:19.720 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.39731.log: -- replaced with /var/log/ceph/cluster1-client.mirror.50796.log.gz 2024-09-18T01:54:19.720 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39800.log 2024-09-18T01:54:19.720 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64180.log 2024-09-18T01:54:19.720 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40260.log: /var/log/ceph/cluster2-client.mirror.39800.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54075.log 2024-09-18T01:54:19.720 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65795.log 2024-09-18T01:54:19.725 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54075.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54075.log.gz 2024-09-18T01:54:19.725 INFO:teuthology.orchestra.run.smithi115.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40260.log.gz 2024-09-18T01:54:19.725 INFO:teuthology.orchestra.run.smithi115.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39731.log.gz 2024-09-18T01:54:19.725 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62128.log 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65795.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64180.log.gz 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65795.log.gz 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63312.log 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62128.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27588.log 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72736.log 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph.tmp-client.admin.27588.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27588.log.gz 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:gzip 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62128.log.gz 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42294.log 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63312.log: /var/log/ceph/cluster2-client.admin.72736.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72736.log.gz 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42670.log 2024-09-18T01:54:19.726 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42294.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42294.log.gz 2024-09-18T01:54:19.751 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67232.log 2024-09-18T01:54:19.751 INFO:teuthology.orchestra.run.smithi115.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63312.log.gz 2024-09-18T01:54:19.751 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37498.log 2024-09-18T01:54:19.751 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67232.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42670.log.gz 2024-09-18T01:54:19.751 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67232.log.gz 2024-09-18T01:54:19.751 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61067.log 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69224.log 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.37498.log: /var/log/ceph/cluster1-client.mirror.61067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71879.log 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69224.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61067.log.gz 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39132.log 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71879.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71879.log.gz 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65131.log 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69224.log.gz 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.39132.log: /var/log/ceph/cluster2-client.mirror.64904.log 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63540.log 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64904.log: gzip 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65131.log.gz 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59443.log 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64904.log.gz 2024-09-18T01:54:19.752 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.63540.log: -- /var/log/ceph/cluster2-client.mirror.41679.log 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63540.log.gz 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59443.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70956.log 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41679.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59443.log.gz 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.admin.37498.log.gz 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41679.log.gz 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38859.log 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39132.log.gz 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72336.log 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70956.log: /var/log/ceph/cluster1-client.mirror.38859.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50953.log 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70956.log.gz 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.72336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72336.log.gz 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66535.log 2024-09-18T01:54:19.753 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49760.log 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66535.log: /var/log/ceph/cluster2-client.mirror.50953.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50953.log.gz 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38859.log.gz 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66535.log.gz 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39442.log 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43704.log 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49760.log: /var/log/ceph/cluster1-client.mirror.39442.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49760.log.gz 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61383.log 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40665.log 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60694.log 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40665.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61383.log.gz 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40665.log.gz 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60694.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60694.log.gz 2024-09-18T01:54:19.754 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52451.log 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40826.log 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43704.log.gz 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52451.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47283.log 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40826.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52451.log.gz 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41751.log 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47283.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47283.log.gz 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55132.log 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.41751.log: --verbose -- /var/log/ceph/cluster1-client.mirror.50242.log 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41751.log.gz 2024-09-18T01:54:19.755 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55132.log: gzip -5 --verbose 93.8%/var/log/ceph/cluster1-client.mirror.50242.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47159.log.gz 2024-09-18T01:54:19.756 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40826.log.gz 2024-09-18T01:54:19.756 INFO:teuthology.orchestra.run.smithi115.stderr: -- 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50242.log.gz 2024-09-18T01:54:19.756 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55132.log.gz 2024-09-18T01:54:19.756 INFO:teuthology.orchestra.run.smithi115.stderr: /var/log/ceph/cluster1-client.mirror.43724.log 2024-09-18T01:54:19.756 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53497.log 2024-09-18T01:54:19.756 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55799.log 2024-09-18T01:54:19.756 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62010.log 2024-09-18T01:54:19.819 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55799.log: 89.6% 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53497.log.gz 2024-09-18T01:54:19.820 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55799.log.gz 2024-09-18T01:54:19.820 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45282.log 2024-09-18T01:54:19.820 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39158.log 2024-09-18T01:54:19.820 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45282.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62010.log.gz 2024-09-18T01:54:19.820 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56973.log 2024-09-18T01:54:19.820 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.39158.log: -- /var/log/ceph/cluster1-client.mirror.56904.log 2024-09-18T01:54:19.820 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56973.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43724.log.gz 2024-09-18T01:54:19.820 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37693.log 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56904.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56973.log.gz 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56904.log.gz 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65544.log 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57332.log 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35700.log 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62276.log 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35700.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35700.log.gz 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55433.log 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62276.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57332.log.gz 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65544.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.37693.log.gz 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:19.821 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53316.log 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62276.log.gz 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46607.log 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55433.log.gz 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58504.log 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53316.log: /var/log/ceph/cluster2-client.mirror.46607.log: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56213.log 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46607.log.gz 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53316.log.gz 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44025.log 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61906.log 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.44025.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56213.log.gz 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42472.log 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61906.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58504.log.gz 2024-09-18T01:54:19.822 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47049.log 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.42472.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61906.log.gz 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59395.log 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47049.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47049.log.gz 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38468.log 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59395.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59395.log.gz 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58943.log 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.38468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35535.log 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42472.log.gz 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58943.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44025.log.gz 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60790.log 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35535.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58943.log.gz 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35535.log.gz 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44876.log 2024-09-18T01:54:19.823 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65898.log 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60790.log.gz 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38996.log 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43400.log 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.38996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70794.log 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43400.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43400.log.gz 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39639.log 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48253.log.gz 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65898.log.gz 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44876.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44876.log.gz 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32287.log 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70794.log: /var/log/ceph/cluster2-client.mirror.39639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65774.log 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70794.log.gz 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41659.log 2024-09-18T01:54:19.824 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.32287.log: /var/log/ceph/cluster2-client.mirror.65774.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32287.log.gz 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54532.log 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54025.log 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54532.log: /var/log/ceph/cluster2-client.mirror.41659.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54532.log.gz 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61248.log 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41659.log.gz/var/log/ceph/cluster2-client.mirror.54025.log: 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39639.log.gz 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54025.log.gz 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr: /var/log/ceph/cluster1-client.mirror.66422.log 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61248.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65774.log.gz 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39262.log 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61248.log.gz/var/log/ceph/cluster1-client.mirror.66422.log: 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66422.log.gz 2024-09-18T01:54:19.825 INFO:teuthology.orchestra.run.smithi115.stderr: -- /var/log/ceph/cluster2-client.mirror.65754.log 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39464.log 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41964.log 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65754.log.gz 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54372.log 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64718.log 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39262.log.gz 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41964.log.gz 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72652.log 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54372.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39593.log 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64718.log: /var/log/ceph/cluster2-client.admin.72652.log: gzip -5 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54372.log.gz 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr: --verbose 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72652.log.gz 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64718.log.gz 2024-09-18T01:54:19.826 INFO:teuthology.orchestra.run.smithi115.stderr: -- /var/log/ceph/cluster2-client.mirror.44220.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66265.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:gzip/var/log/ceph/cluster2-client.mirror.44220.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54879.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50218.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66265.log.gz 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42198.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54879.log.gz 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62300.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42198.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50218.log.gz 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42198.log.gz 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39707.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71374.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62300.log.gz 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53685.log 2024-09-18T01:54:19.827 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43876.log 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71374.log.gz 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62589.log 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53685.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53685.log.gz 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62589.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44220.log.gz 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62589.log.gz 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr: -- /var/log/ceph/cluster2-client.mirror.45695.log 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47698.log 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45695.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39707.log.gz 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35484.log 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47698.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47698.log.gz 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40781.log 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.admin.35484.log: /var/log/ceph/cluster2-client.mirror.48769.log 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62710.log 2024-09-18T01:54:19.828 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40781.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40781.log.gz 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48769.log: 93.5% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43876.log.gz 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48769.log.gz 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50820.log 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49989.log 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62710.log.gz 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50519.log 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49989.log: 87.7% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50820.log.gz 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.49989.log.gz 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49405.log 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63085.log 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49405.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49405.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64924.log 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50519.log: /var/log/ceph/cluster2-client.mirror.63085.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50519.log.gz 2024-09-18T01:54:19.829 INFO:teuthology.orchestra.run.smithi115.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43156.log 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63085.log.gz 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43595.log 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43156.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43156.log.gz 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69351.log 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60513.log 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43595.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64924.log.gz 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62208.log 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60513.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60513.log.gz 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69351.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69351.log.gz 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67322.log 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64552.log 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62208.log.gz 2024-09-18T01:54:19.830 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67322.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53866.log 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67322.log.gz 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52013.log 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53866.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43595.log.gz 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64552.log.gz 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53866.log.gz 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47463.log 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55243.log 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52013.log.gz 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56233.log 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55243.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47463.log.gz 2024-09-18T01:54:19.831 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52338.log 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56233.log: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36709.log 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55243.log.gz 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52338.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39158.log.gz 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52338.log.gz 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37549.log 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69835.log 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.36709.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56233.log.gz 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52401.log 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37549.log: /var/log/ceph/cluster2-client.mirror.69835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64160.log 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37276.log 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64160.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52401.log.gz 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65317.log 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr: 88.2%/var/log/ceph/cluster1-client.admin.37276.log: -- replaced with /var/log/ceph/cluster2-client.mirror.64160.log.gz 2024-09-18T01:54:19.832 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72050.log 2024-09-18T01:54:19.833 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65317.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36709.log.gz 2024-09-18T01:54:19.833 INFO:teuthology.orchestra.run.smithi115.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65317.log.gz 2024-09-18T01:54:19.833 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70535.log 2024-09-18T01:54:19.833 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.72050.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72050.log.gz 2024-09-18T01:54:19.833 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50772.log 2024-09-18T01:54:19.837 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70535.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37549.log.gz 2024-09-18T01:54:19.837 INFO:teuthology.orchestra.run.smithi115.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.37276.log.gz 2024-09-18T01:54:19.837 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52359.log 2024-09-18T01:54:19.837 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50772.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50772.log.gz 2024-09-18T01:54:19.837 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54951.log 2024-09-18T01:54:19.838 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52359.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.35484.log.gz 2024-09-18T01:54:19.838 INFO:teuthology.orchestra.run.smithi115.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52359.log.gz 2024-09-18T01:54:19.838 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61108.log 2024-09-18T01:54:19.838 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54951.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54951.log.gz 2024-09-18T01:54:19.838 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42358.log 2024-09-18T01:54:19.838 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61108.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61108.log.gz 2024-09-18T01:54:19.841 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62058.log 2024-09-18T01:54:19.854 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42358.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71789.log 2024-09-18T01:54:19.855 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62058.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62058.log.gz 2024-09-18T01:54:19.861 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54602.log 2024-09-18T01:54:19.874 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71789.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52854.log 2024-09-18T01:54:19.874 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71789.log.gz 2024-09-18T01:54:19.875 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54602.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54602.log.gz 2024-09-18T01:54:19.879 INFO:teuthology.orchestra.run.smithi115.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42358.log.gz 2024-09-18T01:54:19.880 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41544.log 2024-09-18T01:54:19.880 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52854.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52854.log.gz 2024-09-18T01:54:19.885 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55727.log 2024-09-18T01:54:19.885 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41544.log: 86.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41544.log.gz 2024-09-18T01:54:19.885 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40757.log 2024-09-18T01:54:19.894 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55566.log 2024-09-18T01:54:19.898 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40757.log: 88.5% 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55727.log.gz 2024-09-18T01:54:19.898 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40757.log.gz 2024-09-18T01:54:19.898 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63395.log 2024-09-18T01:54:19.899 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.55566.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55566.log.gz 2024-09-18T01:54:19.901 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72694.log 2024-09-18T01:54:19.907 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63395.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63395.log.gz 2024-09-18T01:54:19.907 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42896.log 2024-09-18T01:54:19.907 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72694.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72694.log.gz 2024-09-18T01:54:19.910 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40414.log 2024-09-18T01:54:19.916 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42896.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42896.log.gz 2024-09-18T01:54:19.916 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40188.log 2024-09-18T01:54:19.916 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55638.log 2024-09-18T01:54:19.919 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53384.log 2024-09-18T01:54:19.922 INFO:teuthology.orchestra.run.smithi115.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40188.log.gz 2024-09-18T01:54:19.923 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55638.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55638.log.gz 2024-09-18T01:54:19.925 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66220.log 2024-09-18T01:54:19.925 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53384.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53384.log.gz 2024-09-18T01:54:19.925 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45223.log 2024-09-18T01:54:19.928 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66220.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66220.log.gz 2024-09-18T01:54:19.934 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67525.log 2024-09-18T01:54:19.934 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45223.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45223.log.gz 2024-09-18T01:54:19.934 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45062.log 2024-09-18T01:54:19.937 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67525.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67525.log.gz 2024-09-18T01:54:19.942 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47012.log 2024-09-18T01:54:19.943 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45062.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45062.log.gz 2024-09-18T01:54:19.943 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44426.log 2024-09-18T01:54:19.951 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47012.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47012.log.gz 2024-09-18T01:54:19.951 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54785.log 2024-09-18T01:54:19.952 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45482.log 2024-09-18T01:54:19.955 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54785.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54785.log.gz 2024-09-18T01:54:19.961 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44426.log.gz 2024-09-18T01:54:19.971 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67007.log 2024-09-18T01:54:19.972 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45482.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45482.log.gz 2024-09-18T01:54:19.972 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59370.log 2024-09-18T01:54:19.972 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67007.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67007.log.gz 2024-09-18T01:54:19.972 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50841.log 2024-09-18T01:54:19.978 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59370.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59370.log.gz 2024-09-18T01:54:19.978 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61501.log 2024-09-18T01:54:19.987 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72946.log 2024-09-18T01:54:19.995 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61501.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61501.log.gz 2024-09-18T01:54:19.995 INFO:teuthology.orchestra.run.smithi115.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50841.log.gz 2024-09-18T01:54:19.995 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.44930.log 2024-09-18T01:54:19.995 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71689.log 2024-09-18T01:54:19.999 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.0.44930.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39464.log.gz 2024-09-18T01:54:20.000 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.44930.log.gz 2024-09-18T01:54:20.004 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58805.log 2024-09-18T01:54:20.004 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.71689.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71689.log.gz 2024-09-18T01:54:20.004 INFO:teuthology.orchestra.run.smithi115.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72946.log.gz 2024-09-18T01:54:20.004 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66850.log 2024-09-18T01:54:20.005 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58805.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58805.log.gz 2024-09-18T01:54:20.005 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36749.log 2024-09-18T01:54:20.005 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66850.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66850.log.gz 2024-09-18T01:54:20.006 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48709.log 2024-09-18T01:54:20.006 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.36749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65255.log 2024-09-18T01:54:20.009 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48709.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39593.log.gz 2024-09-18T01:54:20.009 INFO:teuthology.orchestra.run.smithi115.stderr: 74.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36749.log.gz 2024-09-18T01:54:20.009 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48709.log.gz 2024-09-18T01:54:20.009 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70847.log 2024-09-18T01:54:20.011 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65255.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65255.log.gz 2024-09-18T01:54:20.011 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50061.log 2024-09-18T01:54:20.011 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70847.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70847.log.gz 2024-09-18T01:54:20.012 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62981.log 2024-09-18T01:54:20.015 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50061.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50061.log.gz 2024-09-18T01:54:20.016 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39779.log 2024-09-18T01:54:20.016 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62981.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62981.log.gz 2024-09-18T01:54:20.018 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70455.log 2024-09-18T01:54:20.018 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47740.log 2024-09-18T01:54:20.019 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70455.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39779.log.gz 2024-09-18T01:54:20.022 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39442.log.gz 2024-09-18T01:54:20.024 INFO:teuthology.orchestra.run.smithi115.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70455.log.gz 2024-09-18T01:54:20.024 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50700.log 2024-09-18T01:54:20.024 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47740.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49380.log.gz 2024-09-18T01:54:20.030 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40414.log.gz 2024-09-18T01:54:20.034 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58456.log 2024-09-18T01:54:20.034 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50700.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50700.log.gz 2024-09-18T01:54:20.048 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67412.log 2024-09-18T01:54:20.049 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58456.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58456.log.gz 2024-09-18T01:54:20.049 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45465.log 2024-09-18T01:54:20.050 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67412.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67412.log.gz 2024-09-18T01:54:20.051 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.70207.log 2024-09-18T01:54:20.052 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45465.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45465.log.gz 2024-09-18T01:54:20.052 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67682.log 2024-09-18T01:54:20.053 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.0.70207.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49054.log.gz 2024-09-18T01:54:20.053 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46590.log 2024-09-18T01:54:20.053 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67682.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67682.log.gz 2024-09-18T01:54:20.054 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63726.log 2024-09-18T01:54:20.055 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46590.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46590.log.gz 2024-09-18T01:54:20.057 INFO:teuthology.orchestra.run.smithi115.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.70207.log.gz 2024-09-18T01:54:20.057 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66064.log 2024-09-18T01:54:20.062 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63726.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63726.log.gz 2024-09-18T01:54:20.062 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62690.log 2024-09-18T01:54:20.063 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71809.log 2024-09-18T01:54:20.063 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62690.log: /var/log/ceph/cluster2-client.mirror.66064.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66064.log.gz 2024-09-18T01:54:20.063 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40329.log 2024-09-18T01:54:20.063 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62690.log.gz 2024-09-18T01:54:20.063 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48492.log 2024-09-18T01:54:20.063 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71809.log.gz 2024-09-18T01:54:20.063 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62610.log 2024-09-18T01:54:20.064 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48492.log: /var/log/ceph/cluster2-client.mirror.40329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60674.log 2024-09-18T01:54:20.081 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.62610.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62610.log.gz 2024-09-18T01:54:20.082 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60674.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60674.log.gz 2024-09-18T01:54:20.083 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57677.log 2024-09-18T01:54:20.084 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48995.log 2024-09-18T01:54:20.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57677.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57677.log.gz 2024-09-18T01:54:20.085 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48995.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48995.log.gz 2024-09-18T01:54:20.087 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48550.log 2024-09-18T01:54:20.090 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69289.log 2024-09-18T01:54:20.090 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48550.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48550.log.gz 2024-09-18T01:54:20.090 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52971.log 2024-09-18T01:54:20.090 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69289.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69289.log.gz 2024-09-18T01:54:20.091 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52971.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47946.log 2024-09-18T01:54:20.091 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.38996.log.gz 2024-09-18T01:54:20.093 INFO:teuthology.orchestra.run.smithi115.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52971.log.gz 2024-09-18T01:54:20.093 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60055.log 2024-09-18T01:54:20.094 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47946.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47946.log.gz 2024-09-18T01:54:20.095 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60055.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60055.log.gz 2024-09-18T01:54:20.095 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39800.log.gz 2024-09-18T01:54:20.095 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48689.log 2024-09-18T01:54:20.096 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37361.log 2024-09-18T01:54:20.097 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48689.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48689.log.gz 2024-09-18T01:54:20.097 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60007.log 2024-09-18T01:54:20.099 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.37361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58364.log 2024-09-18T01:54:20.099 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.60007.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60007.log.gz 2024-09-18T01:54:20.099 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52543.log 2024-09-18T01:54:20.102 INFO:teuthology.orchestra.run.smithi115.stderr: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37361.log.gz 2024-09-18T01:54:20.104 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58364.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58364.log.gz 2024-09-18T01:54:20.112 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32460.log 2024-09-18T01:54:20.112 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52543.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52543.log.gz 2024-09-18T01:54:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43264.log 2024-09-18T01:54:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.32460.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32460.log.gz 2024-09-18T01:54:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70328.log 2024-09-18T01:54:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.59076.log 2024-09-18T01:54:20.127 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70328.log: 87.2% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69835.log.gz 2024-09-18T01:54:20.131 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70328.log.gz 2024-09-18T01:54:20.131 INFO:teuthology.orchestra.run.smithi115.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43264.log.gz 2024-09-18T01:54:20.131 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66287.log 2024-09-18T01:54:20.131 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.0.59076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46893.log 2024-09-18T01:54:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66287.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66287.log.gz 2024-09-18T01:54:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58987.log 2024-09-18T01:54:20.141 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68020.log 2024-09-18T01:54:20.143 INFO:teuthology.orchestra.run.smithi115.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.59076.log.gz 2024-09-18T01:54:20.144 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58987.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58987.log.gz 2024-09-18T01:54:20.146 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45653.log 2024-09-18T01:54:20.147 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68020.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68020.log.gz 2024-09-18T01:54:20.147 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70435.log 2024-09-18T01:54:20.151 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45653.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45653.log.gz 2024-09-18T01:54:20.151 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47929.log 2024-09-18T01:54:20.153 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70435.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70435.log.gz 2024-09-18T01:54:20.153 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42968.log 2024-09-18T01:54:20.153 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47929.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47929.log.gz 2024-09-18T01:54:20.160 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58323.log 2024-09-18T01:54:20.160 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.42968.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42968.log.gz 2024-09-18T01:54:20.162 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55409.log 2024-09-18T01:54:20.162 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58323.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58323.log.gz 2024-09-18T01:54:20.162 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65602.log 2024-09-18T01:54:20.164 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55409.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55409.log.gz 2024-09-18T01:54:20.164 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61477.log 2024-09-18T01:54:20.165 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.65602.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65602.log.gz 2024-09-18T01:54:20.165 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70584.log 2024-09-18T01:54:20.166 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61477.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61477.log.gz 2024-09-18T01:54:20.166 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47538.log 2024-09-18T01:54:20.169 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70584.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70584.log.gz 2024-09-18T01:54:20.170 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67075.log 2024-09-18T01:54:20.170 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47538.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47538.log.gz 2024-09-18T01:54:20.173 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72384.log 2024-09-18T01:54:20.173 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67075.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67075.log.gz 2024-09-18T01:54:20.174 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57085.log 2024-09-18T01:54:20.174 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.72384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72384.log.gz 2024-09-18T01:54:20.174 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32599.log 2024-09-18T01:54:20.174 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57085.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57085.log.gz 2024-09-18T01:54:20.178 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51853.log 2024-09-18T01:54:20.178 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.32599.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32599.log.gz 2024-09-18T01:54:20.178 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37293.log 2024-09-18T01:54:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51853.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51853.log.gz 2024-09-18T01:54:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68042.log 2024-09-18T01:54:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.37293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46524.log 2024-09-18T01:54:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68042.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68042.log.gz 2024-09-18T01:54:20.182 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46834.log 2024-09-18T01:54:20.184 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46524.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46524.log.gz 2024-09-18T01:54:20.184 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59076.log 2024-09-18T01:54:20.185 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46834.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46834.log.gz 2024-09-18T01:54:20.187 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63250.log 2024-09-18T01:54:20.187 INFO:teuthology.orchestra.run.smithi115.stderr: 66.2% -- replaced with /var/log/ceph/cluster1-client.admin.37293.log.gz 2024-09-18T01:54:20.187 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59076.log: 82.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59076.log.gz 2024-09-18T01:54:20.191 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46951.log 2024-09-18T01:54:20.195 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63250.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63250.log.gz 2024-09-18T01:54:20.195 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50314.log 2024-09-18T01:54:20.196 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46951.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46951.log.gz 2024-09-18T01:54:20.196 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65565.log 2024-09-18T01:54:20.198 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50314.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50314.log.gz 2024-09-18T01:54:20.198 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54489.log 2024-09-18T01:54:20.198 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45695.log.gz 2024-09-18T01:54:20.199 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65565.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65565.log.gz 2024-09-18T01:54:20.199 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40168.log 2024-09-18T01:54:20.202 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54489.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54489.log.gz 2024-09-18T01:54:20.202 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51049.log 2024-09-18T01:54:20.203 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66557.log 2024-09-18T01:54:20.206 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51049.log: gzip 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51049.log.gz 2024-09-18T01:54:20.206 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.35701.log 2024-09-18T01:54:20.206 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41799.log 2024-09-18T01:54:20.206 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35701.log: /var/log/ceph/cluster1-client.mirror.66557.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35701.log.gz 2024-09-18T01:54:20.206 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66557.log.gz 2024-09-18T01:54:20.206 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47521.log 2024-09-18T01:54:20.206 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45303.log 2024-09-18T01:54:20.206 INFO:teuthology.orchestra.run.smithi115.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41799.log.gz 2024-09-18T01:54:20.209 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45303.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40168.log.gz 2024-09-18T01:54:20.209 INFO:teuthology.orchestra.run.smithi115.stderr: 77.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45303.log.gz 2024-09-18T01:54:20.209 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47521.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47521.log.gz 2024-09-18T01:54:20.209 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64428.log 2024-09-18T01:54:20.209 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57627.log 2024-09-18T01:54:20.212 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64428.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64428.log.gz 2024-09-18T01:54:20.212 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64098.log 2024-09-18T01:54:20.213 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.57627.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57627.log.gz 2024-09-18T01:54:20.214 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59252.log 2024-09-18T01:54:20.215 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64098.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64098.log.gz 2024-09-18T01:54:20.215 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61986.log 2024-09-18T01:54:20.217 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.59252.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59252.log.gz 2024-09-18T01:54:20.217 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59730.log 2024-09-18T01:54:20.218 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61986.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61986.log.gz 2024-09-18T01:54:20.218 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64656.log 2024-09-18T01:54:20.219 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59730.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59730.log.gz 2024-09-18T01:54:20.219 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44005.log 2024-09-18T01:54:20.221 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64656.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64656.log.gz 2024-09-18T01:54:20.221 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50471.log 2024-09-18T01:54:20.222 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56255.log 2024-09-18T01:54:20.222 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50471.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50471.log.gz 2024-09-18T01:54:20.222 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52567.log 2024-09-18T01:54:20.224 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56255.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56255.log.gz 2024-09-18T01:54:20.228 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45282.log.gz 2024-09-18T01:54:20.228 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72408.log 2024-09-18T01:54:20.228 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44005.log.gz 2024-09-18T01:54:20.228 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43244.log 2024-09-18T01:54:20.228 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72408.log: 87.4% 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52567.log.gz 2024-09-18T01:54:20.228 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.72408.log.gz 2024-09-18T01:54:20.228 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62851.log 2024-09-18T01:54:20.229 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58709.log 2024-09-18T01:54:20.232 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43244.log: /var/log/ceph/cluster2-client.mirror.62851.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62851.log.gz 2024-09-18T01:54:20.232 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35403.log 2024-09-18T01:54:20.232 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58709.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42498.log 2024-09-18T01:54:20.232 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58709.log.gz 2024-09-18T01:54:20.233 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35403.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35403.log.gz 2024-09-18T01:54:20.233 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39885.log 2024-09-18T01:54:20.233 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42498.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42498.log.gz 2024-09-18T01:54:20.235 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48849.log 2024-09-18T01:54:20.235 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70370.log 2024-09-18T01:54:20.235 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48849.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43244.log.gz 2024-09-18T01:54:20.239 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39399.log 2024-09-18T01:54:20.239 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70370.log: gzip -5 --verbose 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48849.log.gz 2024-09-18T01:54:20.239 INFO:teuthology.orchestra.run.smithi115.stderr: -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70370.log.gz 2024-09-18T01:54:20.239 INFO:teuthology.orchestra.run.smithi115.stderr: /var/log/ceph/cluster1-client.mirror.1.47049.log 2024-09-18T01:54:20.240 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43446.log 2024-09-18T01:54:20.240 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.1.47049.log: /var/log/ceph/cluster1-client.mirror.39399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37532.log 2024-09-18T01:54:20.240 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45502.log 2024-09-18T01:54:20.240 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37532.log: /var/log/ceph/cluster2-client.mirror.43446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51367.log 2024-09-18T01:54:20.241 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45502.log: 87.5% 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39885.log.gz 2024-09-18T01:54:20.241 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45502.log.gz 2024-09-18T01:54:20.242 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46507.log 2024-09-18T01:54:20.243 INFO:teuthology.orchestra.run.smithi115.stderr: 54.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37532.log.gz 2024-09-18T01:54:20.243 INFO:teuthology.orchestra.run.smithi115.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43446.log.gz 2024-09-18T01:54:20.246 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46507.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46507.log.gz 2024-09-18T01:54:20.246 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.47049.log.gz 2024-09-18T01:54:20.246 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61433.log 2024-09-18T01:54:20.246 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69793.log 2024-09-18T01:54:20.246 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60835.log 2024-09-18T01:54:20.248 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.61433.log: /var/log/ceph/cluster2-client.mirror.69793.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51367.log.gz 2024-09-18T01:54:20.248 INFO:teuthology.orchestra.run.smithi115.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61433.log.gz 2024-09-18T01:54:20.248 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66737.log 2024-09-18T01:54:20.249 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.60835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69330.log 2024-09-18T01:54:20.249 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66737.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66737.log.gz 2024-09-18T01:54:20.249 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46384.log 2024-09-18T01:54:20.250 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69330.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60835.log.gz 2024-09-18T01:54:20.250 INFO:teuthology.orchestra.run.smithi115.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54835.log 2024-09-18T01:54:20.250 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69793.log.gz 2024-09-18T01:54:20.250 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61605.log 2024-09-18T01:54:20.250 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54835.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46384.log.gz 2024-09-18T01:54:20.253 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53083.log 2024-09-18T01:54:20.253 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.61605.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54835.log.gz 2024-09-18T01:54:20.253 INFO:teuthology.orchestra.run.smithi115.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69330.log.gz 2024-09-18T01:54:20.253 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61605.log.gzgzip 2024-09-18T01:54:20.253 INFO:teuthology.orchestra.run.smithi115.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49871.log 2024-09-18T01:54:20.254 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.53083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70305.log 2024-09-18T01:54:20.254 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67907.log 2024-09-18T01:54:20.254 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70305.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49871.log.gz 2024-09-18T01:54:20.254 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70305.log.gz 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53083.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57133.log 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72140.log 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67907.log: /var/log/ceph/cluster1-client.mirror.57133.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67907.log.gz 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57133.log.gz 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46023.log 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.61752.log 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.72140.log: 0.0%/var/log/ceph/cluster2-client.mirror.46023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41276.log 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46023.log.gz 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72140.log.gz 2024-09-18T01:54:20.259 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.0.61752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49012.log 2024-09-18T01:54:20.260 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.36866.log 2024-09-18T01:54:20.260 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.41276.log: /var/log/ceph/cluster2-client.mirror.49012.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41276.log.gz 2024-09-18T01:54:20.260 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39379.log 2024-09-18T01:54:20.260 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.3.36866.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49012.log.gz 2024-09-18T01:54:20.260 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63870.log 2024-09-18T01:54:20.261 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40016.log 2024-09-18T01:54:20.263 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63870.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63870.log.gz 2024-09-18T01:54:20.263 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56002.log 2024-09-18T01:54:20.263 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40016.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.61752.log.gz 2024-09-18T01:54:20.264 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49595.log 2024-09-18T01:54:20.265 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56002.log.gz 2024-09-18T01:54:20.269 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49595.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49595.log.gz 2024-09-18T01:54:20.269 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49276.log 2024-09-18T01:54:20.269 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68132.log 2024-09-18T01:54:20.269 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.49276.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49276.log.gz 2024-09-18T01:54:20.269 INFO:teuthology.orchestra.run.smithi115.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40016.log.gz 2024-09-18T01:54:20.270 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40053.log 2024-09-18T01:54:20.270 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42315.log 2024-09-18T01:54:20.270 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68132.log.gz 2024-09-18T01:54:20.270 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39659.log 2024-09-18T01:54:20.270 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46973.log 2024-09-18T01:54:20.270 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72112.log 2024-09-18T01:54:20.272 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39659.log: /var/log/ceph/cluster2-client.mirror.72112.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39659.log.gz 2024-09-18T01:54:20.273 INFO:teuthology.orchestra.run.smithi115.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72112.log.gz 2024-09-18T01:54:20.275 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46973.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42315.log.gz 2024-09-18T01:54:20.276 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48433.log 2024-09-18T01:54:20.276 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-18T01:54:20.276 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48433.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48433.log.gz 2024-09-18T01:54:20.279 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72778.log 2024-09-18T01:54:20.285 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2.log: 87.6% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-18T01:54:20.285 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70736.log 2024-09-18T01:54:20.286 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.72778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72778.log.gz 2024-09-18T01:54:20.286 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60284.log 2024-09-18T01:54:20.289 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70736.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70736.log.gz 2024-09-18T01:54:20.290 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46305.log 2024-09-18T01:54:20.290 INFO:teuthology.orchestra.run.smithi115.stderr: 95.3%/var/log/ceph/cluster1-client.mirror.60284.log: -- replaced with /var/log/ceph/cluster1-client.mirror.40053.log.gz 2024-09-18T01:54:20.290 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60284.log.gz 2024-09-18T01:54:20.291 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65981.log 2024-09-18T01:54:20.291 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46305.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46305.log.gz 2024-09-18T01:54:20.292 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36079.log 2024-09-18T01:54:20.293 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38931.log 2024-09-18T01:54:20.293 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.36079.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36079.log.gz 2024-09-18T01:54:20.294 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50124.log 2024-09-18T01:54:20.294 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.38931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36789.log 2024-09-18T01:54:20.295 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.50124.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50124.log.gz 2024-09-18T01:54:20.296 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56699.log 2024-09-18T01:54:20.299 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.36789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39683.log 2024-09-18T01:54:20.299 INFO:teuthology.orchestra.run.smithi115.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65981.log.gz 2024-09-18T01:54:20.299 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39755.log 2024-09-18T01:54:20.299 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39683.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56699.log.gz 2024-09-18T01:54:20.299 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65152.log 2024-09-18T01:54:20.299 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.39755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42650.log 2024-09-18T01:54:20.301 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65152.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39683.log.gz 2024-09-18T01:54:20.301 INFO:teuthology.orchestra.run.smithi115.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65152.log.gz 2024-09-18T01:54:20.301 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51712.log 2024-09-18T01:54:20.302 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.51712.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51712.log.gz 2024-09-18T01:54:20.304 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42650.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42650.log.gz 2024-09-18T01:54:20.304 INFO:teuthology.orchestra.run.smithi115.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.47485.log 2024-09-18T01:54:20.304 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45673.log 2024-09-18T01:54:20.304 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47485.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39755.log.gz 2024-09-18T01:54:20.306 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51531.log 2024-09-18T01:54:20.306 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45673.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45673.log.gz 2024-09-18T01:54:20.307 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63292.log 2024-09-18T01:54:20.307 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51531.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38931.log.gz 2024-09-18T01:54:20.307 INFO:teuthology.orchestra.run.smithi115.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51531.log.gz 2024-09-18T01:54:20.308 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57605.log 2024-09-18T01:54:20.309 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63292.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63292.log.gz 2024-09-18T01:54:20.309 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.36921.log 2024-09-18T01:54:20.310 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.57605.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57605.log.gz 2024-09-18T01:54:20.312 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53637.log 2024-09-18T01:54:20.313 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.1.36921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70977.log 2024-09-18T01:54:20.313 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53637.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53637.log.gz 2024-09-18T01:54:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.70977.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70977.log.gz 2024-09-18T01:54:20.316 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62148.log 2024-09-18T01:54:20.316 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38955.log 2024-09-18T01:54:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62148.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39379.log.gz 2024-09-18T01:54:20.326 INFO:teuthology.orchestra.run.smithi115.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62148.log.gz 2024-09-18T01:54:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67187.log 2024-09-18T01:54:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.38955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67435.log 2024-09-18T01:54:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67187.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67187.log.gz 2024-09-18T01:54:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54256.log 2024-09-18T01:54:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67435.log: 85.2% -- replaced with /var/log/ceph/cluster1-client.mirror.67435.log.gz 2024-09-18T01:54:20.334 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38955.log.gz 2024-09-18T01:54:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66242.log 2024-09-18T01:54:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54256.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54256.log.gz 2024-09-18T01:54:20.341 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55614.log 2024-09-18T01:54:20.342 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66242.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66242.log.gz 2024-09-18T01:54:20.342 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37656.log 2024-09-18T01:54:20.343 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55614.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55614.log.gz 2024-09-18T01:54:20.343 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37259.log 2024-09-18T01:54:20.345 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.37656.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37656.log.gz 2024-09-18T01:54:20.352 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67615.log 2024-09-18T01:54:20.353 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.37259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68335.log 2024-09-18T01:54:20.353 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67615.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67615.log.gz 2024-09-18T01:54:20.355 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55221.log 2024-09-18T01:54:20.355 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68335.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68335.log.gz 2024-09-18T01:54:20.355 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54743.log 2024-09-18T01:54:20.356 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.55221.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55221.log.gz 2024-09-18T01:54:20.358 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40329.log.gz 2024-09-18T01:54:20.358 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46285.log 2024-09-18T01:54:20.359 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.54743.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.37259.log.gz 2024-09-18T01:54:20.359 INFO:teuthology.orchestra.run.smithi115.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54743.log.gz 2024-09-18T01:54:20.360 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47601.log 2024-09-18T01:54:20.362 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46285.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46285.log.gz 2024-09-18T01:54:20.362 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50543.log 2024-09-18T01:54:20.363 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47601.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47601.log.gz 2024-09-18T01:54:20.363 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53709.log 2024-09-18T01:54:20.363 INFO:teuthology.orchestra.run.smithi115.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.36921.log.gz 2024-09-18T01:54:20.364 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50543.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50543.log.gz 2024-09-18T01:54:20.365 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65090.log 2024-09-18T01:54:20.366 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.53709.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53709.log.gz 2024-09-18T01:54:20.366 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48912.log 2024-09-18T01:54:20.367 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65090.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65090.log.gz 2024-09-18T01:54:20.367 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63230.log 2024-09-18T01:54:20.368 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48912.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48912.log.gz 2024-09-18T01:54:20.368 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51389.log 2024-09-18T01:54:20.370 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63230.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63230.log.gz 2024-09-18T01:54:20.370 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43509.log 2024-09-18T01:54:20.371 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51389.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51389.log.gz 2024-09-18T01:54:20.372 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49468.log 2024-09-18T01:54:20.380 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54143.log 2024-09-18T01:54:20.382 INFO:teuthology.orchestra.run.smithi115.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43509.log.gz 2024-09-18T01:54:20.382 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.49468.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49468.log.gz 2024-09-18T01:54:20.382 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37235.log 2024-09-18T01:54:20.383 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.54143.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54143.log.gz 2024-09-18T01:54:20.385 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64263.log 2024-09-18T01:54:20.385 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.37235.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37235.log.gz 2024-09-18T01:54:20.385 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63333.log 2024-09-18T01:54:20.389 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64263.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64263.log.gz 2024-09-18T01:54:20.389 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50977.log 2024-09-18T01:54:20.390 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63333.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47906.log.gz 2024-09-18T01:54:20.391 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51114.log 2024-09-18T01:54:20.391 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50977.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63333.log.gz 2024-09-18T01:54:20.391 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52242.log 2024-09-18T01:54:20.391 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50977.log.gz 2024-09-18T01:54:20.391 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.51114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35952.log 2024-09-18T01:54:20.393 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52242.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52242.log.gz 2024-09-18T01:54:20.393 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38815.log 2024-09-18T01:54:20.393 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.35952.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51114.log.gz 2024-09-18T01:54:20.394 INFO:teuthology.orchestra.run.smithi115.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35952.log.gz 2024-09-18T01:54:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64201.log 2024-09-18T01:54:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.38815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67255.log 2024-09-18T01:54:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:gzip/var/log/ceph/cluster2-client.mirror.64201.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50567.log 2024-09-18T01:54:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.67255.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64201.log.gz 2024-09-18T01:54:20.396 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38815.log.gz 2024-09-18T01:54:20.396 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67255.log.gz 2024-09-18T01:54:20.396 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70912.log 2024-09-18T01:54:20.397 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50567.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50567.log.gz 2024-09-18T01:54:20.397 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46690.log 2024-09-18T01:54:20.398 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52471.log 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70912.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70912.log.gz 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48630.log 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.52471.log: 87.4% 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46690.log.gz 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52471.log.gz 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46673.log 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.48630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56928.log 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48630.log.gz 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46673.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46673.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48752.log 2024-09-18T01:54:20.401 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.56928.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56928.log.gz 2024-09-18T01:54:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66175.log 2024-09-18T01:54:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59419.log 2024-09-18T01:54:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.66175.log: /var/log/ceph/cluster2-client.mirror.48752.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48752.log.gz 2024-09-18T01:54:20.404 INFO:teuthology.orchestra.run.smithi115.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66175.log.gz 2024-09-18T01:54:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63064.log 2024-09-18T01:54:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68404.log 2024-09-18T01:54:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59419.log: /var/log/ceph/cluster2-client.mirror.63064.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59419.log.gz 2024-09-18T01:54:20.405 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68290.log 2024-09-18T01:54:20.406 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68404.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68404.log.gz 2024-09-18T01:54:20.406 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40392.log 2024-09-18T01:54:20.406 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.68290.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68290.log.gz 2024-09-18T01:54:20.408 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49195.log 2024-09-18T01:54:20.408 INFO:teuthology.orchestra.run.smithi115.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63064.log.gz 2024-09-18T01:54:20.408 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.40392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44263.log 2024-09-18T01:54:20.408 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.49195.log: -- /var/log/ceph/cluster1-client.mirror.58617.log 2024-09-18T01:54:20.409 INFO:teuthology.orchestra.run.smithi115.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49195.log.gz 2024-09-18T01:54:20.409 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56720.log 2024-09-18T01:54:20.410 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.58617.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58617.log.gz 2024-09-18T01:54:20.415 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62734.log 2024-09-18T01:54:20.415 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.56720.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44263.log.gz 2024-09-18T01:54:20.416 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69908.log 2024-09-18T01:54:20.416 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62734.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62734.log.gz 2024-09-18T01:54:20.419 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64780.log 2024-09-18T01:54:20.419 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.69908.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56720.log.gz 2024-09-18T01:54:20.419 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43084.log 2024-09-18T01:54:20.419 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69908.log.gz 2024-09-18T01:54:20.420 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44686.log 2024-09-18T01:54:20.420 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64780.log: /var/log/ceph/cluster1-client.mirror.43084.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43084.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64780.log.gz 2024-09-18T01:54:20.420 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:54:20.420 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70651.log 2024-09-18T01:54:20.420 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.44686.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44686.log.gz 2024-09-18T01:54:20.421 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52742.log 2024-09-18T01:54:20.421 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39218.log 2024-09-18T01:54:20.423 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.70651.log: /var/log/ceph/cluster2-client.mirror.52742.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70651.log.gz 2024-09-18T01:54:20.423 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52742.log.gz 2024-09-18T01:54:20.423 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60718.log 2024-09-18T01:54:20.425 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.39218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36417.log 2024-09-18T01:54:20.425 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32665.log 2024-09-18T01:54:20.425 INFO:teuthology.orchestra.run.smithi115.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39218.log.gz 2024-09-18T01:54:20.425 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.36417.log: /var/log/ceph/cluster1-client.mirror.60718.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36417.log.gz 2024-09-18T01:54:20.426 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69042.log 2024-09-18T01:54:20.427 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.32665.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32665.log.gz 2024-09-18T01:54:20.427 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60718.log.gz 2024-09-18T01:54:20.427 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43221.log 2024-09-18T01:54:20.428 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.69042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32500.log 2024-09-18T01:54:20.428 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43221.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69042.log.gz 2024-09-18T01:54:20.430 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52926.log 2024-09-18T01:54:20.431 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.admin.32500.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32500.log.gz 2024-09-18T01:54:20.431 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46486.log 2024-09-18T01:54:20.432 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.52926.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52926.log.gz 2024-09-18T01:54:20.432 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42222.log 2024-09-18T01:54:20.433 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.46486.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43221.log.gz 2024-09-18T01:54:20.434 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64387.log 2024-09-18T01:54:20.434 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.42222.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42222.log.gz 2024-09-18T01:54:20.435 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45742.log 2024-09-18T01:54:20.436 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64387.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64387.log.gz 2024-09-18T01:54:20.437 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45742.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45742.log.gz 2024-09-18T01:54:20.442 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62324.log 2024-09-18T01:54:20.442 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42107.log 2024-09-18T01:54:20.442 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62324.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62324.log.gz 2024-09-18T01:54:20.452 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58183.log 2024-09-18T01:54:20.453 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.42107.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42107.log.gz 2024-09-18T01:54:20.453 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47244.log 2024-09-18T01:54:20.453 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.58183.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58183.log.gz 2024-09-18T01:54:20.455 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64966.log 2024-09-18T01:54:20.471 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46893.log.gz 2024-09-18T01:54:20.472 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47426.log 2024-09-18T01:54:20.475 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.64966.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64966.log.gz 2024-09-18T01:54:20.475 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45386.log 2024-09-18T01:54:20.475 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47426.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47426.log.gz 2024-09-18T01:54:20.475 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71831.log 2024-09-18T01:54:20.475 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.45386.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45386.log.gz 2024-09-18T01:54:20.475 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68439.log 2024-09-18T01:54:20.484 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.71831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40619.log 2024-09-18T01:54:20.484 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.68439.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68439.log.gz 2024-09-18T01:54:20.498 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41406.log 2024-09-18T01:54:20.499 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.40619.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40619.log.gz 2024-09-18T01:54:20.504 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70535.log.gz 2024-09-18T01:54:20.508 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50748.log 2024-09-18T01:54:20.509 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.41406.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41406.log.gz 2024-09-18T01:54:20.510 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63168.log 2024-09-18T01:54:20.510 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.50748.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50748.log.gz 2024-09-18T01:54:20.511 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51600.log 2024-09-18T01:54:20.511 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.63168.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63168.log.gz 2024-09-18T01:54:20.512 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72215.log 2024-09-18T01:54:20.514 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.51600.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51600.log.gz 2024-09-18T01:54:20.514 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65028.log 2024-09-18T01:54:20.515 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.72215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72215.log.gz 2024-09-18T01:54:20.515 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47347.log 2024-09-18T01:54:20.515 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.65028.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65028.log.gz 2024-09-18T01:54:20.516 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43661.log 2024-09-18T01:54:20.516 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.47347.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47347.log.gz 2024-09-18T01:54:20.517 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40955.log 2024-09-18T01:54:20.522 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.43661.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43661.log.gz 2024-09-18T01:54:20.523 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62389.log 2024-09-18T01:54:20.524 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.40955.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.40955.log.gz 2024-09-18T01:54:20.524 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68476.log 2024-09-18T01:54:20.526 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.62389.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62389.log.gz 2024-09-18T01:54:20.526 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43180.log 2024-09-18T01:54:20.527 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster2-client.mirror.68476.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68476.log.gz 2024-09-18T01:54:20.527 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59778.log 2024-09-18T01:54:20.528 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.43180.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43180.log.gz 2024-09-18T01:54:20.529 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42081.log 2024-09-18T01:54:20.529 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.mirror.59778.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59778.log.gz 2024-09-18T01:54:20.530 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cluster1-client.admin.42081.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42081.log.gz 2024-09-18T01:54:20.541 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39399.log.gz 2024-09-18T01:54:20.584 INFO:teuthology.orchestra.run.smithi115.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47485.log.gz 2024-09-18T01:54:20.645 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48492.log.gz 2024-09-18T01:54:20.653 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47740.log.gz 2024-09-18T01:54:20.655 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40392.log.gz 2024-09-18T01:54:20.681 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46973.log.gz 2024-09-18T01:54:20.734 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47244.log.gz 2024-09-18T01:54:20.776 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46486.log.gz 2024-09-18T01:54:20.909 INFO:teuthology.orchestra.run.smithi115.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.36866.log.gz 2024-09-18T01:54:21.037 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71831.log.gz 2024-09-18T01:54:21.230 INFO:teuthology.orchestra.run.smithi115.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36789.log.gz 2024-09-18T01:54:21.836 INFO:teuthology.orchestra.run.smithi115.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-18T01:54:24.083 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39040.log.gz 2024-09-18T01:54:24.293 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38468.log.gz 2024-09-18T01:54:24.688 INFO:teuthology.orchestra.run.smithi066.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-18T01:54:24.827 INFO:teuthology.orchestra.run.smithi066.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-18T01:54:25.806 INFO:teuthology.orchestra.run.smithi066.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-18T01:54:25.808 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-09-18T01:54:25.808 INFO:teuthology.orchestra.run.smithi066.stderr:real 0m9.820s 2024-09-18T01:54:25.808 INFO:teuthology.orchestra.run.smithi066.stderr:user 0m27.932s 2024-09-18T01:54:25.808 INFO:teuthology.orchestra.run.smithi066.stderr:sys 0m0.948s 2024-09-18T01:54:25.991 INFO:teuthology.orchestra.run.smithi115.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39907.log.gz 2024-09-18T01:55:45.673 INFO:teuthology.orchestra.run.smithi115.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-18T01:55:48.090 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-18T01:56:13.466 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-18T01:56:13.467 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-18T01:56:13.468 INFO:teuthology.orchestra.run.smithi115.stderr:real 1m57.447s 2024-09-18T01:56:13.468 INFO:teuthology.orchestra.run.smithi115.stderr:user 3m51.061s 2024-09-18T01:56:13.468 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m10.078s 2024-09-18T01:56:13.468 INFO:tasks.ceph:Archiving logs... 2024-09-18T01:56:13.469 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706/remote/smithi066/log 2024-09-18T01:56:13.470 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T01:56:17.293 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706/remote/smithi115/log 2024-09-18T01:56:17.294 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T01:56:52.152 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T01:56:52.164 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T01:56:52.165 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T01:56:52.167 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T01:56:52.397 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T01:56:52.408 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T01:56:52.408 DEBUG:teuthology.orchestra.run.smithi066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T01:56:52.410 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T01:56:52.426 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-09-18T01:56:52.428 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-09-18T01:56:52.431 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T01:56:52.431 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-09-18T01:56:52.431 INFO:teuthology.orchestra.run.smithi066.stdout:^* hv01.front.sepia.ceph.com 3 8 377 10 -317us[ -202us] +/- 43ms 2024-09-18T01:56:52.431 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -1214us[-1214us] +/- 63ms 2024-09-18T01:56:52.432 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 11 +1970us[+2085us] +/- 66ms 2024-09-18T01:56:52.432 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:56:52.460 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T01:56:52.460 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-09-18T01:56:52.460 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv01.front.sepia.ceph.com 3 6 377 18 -355us[ -357us] +/- 43ms 2024-09-18T01:56:52.460 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 -1233us[-1233us] +/- 63ms 2024-09-18T01:56:52.460 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +1924us[+1924us] +/- 66ms 2024-09-18T01:56:52.460 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T01:56:52.461 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T01:56:52.473 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T01:56:52.473 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T01:56:52.484 DEBUG:teuthology.orchestra.run.smithi066:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T01:56:52.555 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T01:56:52.655 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T01:56:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:56:52.687 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T01:56:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:56:52.716 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T01:56:52.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T01:56:52.740 INFO:teuthology.task.internal:Duration was 2832.431460 seconds 2024-09-18T01:56:52.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T01:56:52.752 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T01:56:52.752 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T01:56:52.755 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T01:56:52.795 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T01:56:52.818 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T01:56:53.327 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T01:56:53.327 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi066.front.sepia.ceph.com 2024-09-18T01:56:53.327 DEBUG:teuthology.orchestra.run.smithi066:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T01:56:53.349 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T01:56:53.350 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T01:56:53.372 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T01:56:53.372 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T01:56:53.393 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T01:56:53.441 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T01:56:53.441 DEBUG:teuthology.orchestra.run.smithi066:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T01:56:53.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T01:56:53.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T01:56:53.606 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T01:56:53.607 DEBUG:teuthology.orchestra.run.smithi066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T01:56:53.632 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T01:56:53.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T01:56:53.677 DEBUG:teuthology.orchestra.run.smithi066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T01:56:53.680 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T01:56:53.701 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = core 2024-09-18T01:56:53.733 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2024-09-18T01:56:53.746 DEBUG:teuthology.orchestra.run.smithi066:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T01:56:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:56:53.779 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T01:56:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T01:56:53.805 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T01:56:53.818 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T01:56:53.818 DEBUG:teuthology.misc:Transferring archived files from smithi066:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706/remote/smithi066 2024-09-18T01:56:53.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T01:56:53.853 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909706/remote/smithi115 2024-09-18T01:56:53.854 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T01:56:53.886 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T01:56:53.886 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T01:56:53.890 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T01:56:53.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T01:56:53.950 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T01:56:53.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T01:56:54.041 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T01:56:54.041 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T01:56:54.043 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T01:56:54.056 INFO:teuthology.orchestra.run.smithi066.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 01:56 /home/ubuntu/cephtest 2024-09-18T01:56:54.058 INFO:teuthology.orchestra.run.smithi115.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 01:56 /home/ubuntu/cephtest 2024-09-18T01:56:54.059 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T01:56:54.144 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T01:56:54.172 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 2832.431460380554 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-18T01:56:54.172 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T01:56:54.250 INFO:teuthology.run:pass